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Preface 


The  ITI-ALC  Phase  II  effort  was  sponsored  by  the  United  States  Air  Force  Research  Laboratory 
(AFRL),  Human  Effectiveness  Directorate  (HESR),  Wright  Patterson  Air  Force  Base,  Dayton, 
Ohio.  Continuity  and  understanding  of  the  work  done  for  ITI-ALC  Phase  I  was  facilitated  by  the 
presence  of  the  same  Laboratory  personnel  for  both  phases  of  the  ITI-ALC  effort.  (Phase  I  work 
was  completed  by  Systems  Research  and  Applications  Corporation  (SRA)  in  1995,  and 
generated  several  key  documents  used  as  inputs  to  ITI-ALC  Phase  II  including:  SSI,  etc.). 

Laboratory  personnel  participated  actively  in  all  major  activities  of  the  Phase  II  project.  They 
reviewed  early  prototype  designs,  arranged  logistics,  observed  user  and  contractor  behavior 
during  site  visits,  and  served  as  data  collectors  during  the  field-based  evaluation  activities. 
Individuals  who  represented  AFRL/HESR  included:  Captain  Bob  Hartz,  Lt.  Patrick  Pohle,  Lt. 
Steven  Grace,  Ms.  Barbara  Masquelier,  Dr.  Donald  Thomas,  Ms.  Cheryl  Batchelor,  and  Mr.  Paul 
Faas. 

The  ITI-ALC  Phase  II  effort  began  with  Lockheed  Martin  Information  Systems  (LMIS), 
Orlando,  Florida  as  the  Prime  Contractor  assisted  by  three  subcontractors:  Battelle,  Carnegie 
Mellon  University,  and  Lockheed  Martin  Advanced  Technology  Laboratory  (LM  ATL).  As  the 
project  progressed,  the  statement  of  work  and  resources  available  were  modified  and  both 
Battelle  and  LM  ATL  completed  their  involvement  with  the  effort. 
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1.  ITI-ALC  Phase  II  Summary 


1.1  Introduction 

Phase  II  of  the  ITI-ALC  program  utilized  an  iterative,  user-centered  development 
approach  to  identify  the  PDM  processes  currently  in  use,  propose  technologies  that 
could  be  affordably  used  to  automate  those  processes,  and  implement  prototype 
applications  to  evaluate  those  proposals.  This  approach  provided  the  program's 
subjects  (PDM  mechanics)  with  periodic  opportunities  to  review  and  guide  the 
development  effort  in  directions  that  best  fit  the  environment  and  processes  that  they 
live  with  on  a  daily  basis.  The  team's  frequent  visits  and  interactions  with  the  subjects 
kept  them  interested  and  involved  throughout  the  twenty  months  of  Phase  II. 

Ten  PDM  personnel  participated  in  FBE  #1  in  June  1997.  Nine  of  the  ten  had  no  prior 
computer  experience.  They  inspected  and  tried  out  a  variety  of  advanced  hardware  and 
software,  witnessed  a  demonstration  and  discussed  with  the  ITI-ALC  team  how  these 
devices  and  techniques  could  help  them  spend  more  time  working  on  aircraft.  Based 
on  the  results  of  these  discussions,  we  developed  a  plan  to  demonstrate  and  evaluate 
tools  that  automated,  to  various  degrees,  the  following  PDM  processes: 

•  Aircraft  Evaluation  and  Inventory  (E&l) 

•  Engineering  Assistance  Request  (202  Form) 

•  Electronic  Identification  and  Signoff 

•  Electronic  Technical  Order  (TO)  Delivery 

•  Operations  Check 

•  PDM  Planning  and  Scheduling 

•  PDM/Backshop  Schedule  Coordination 

•  Backshop  Routing  and  Tracking 

As  the  program  progressed,  the  initial  thirty-six  month  period  of  performance  was 
reduced  to  twenty  months  due  to  funding  cuts.  While  the  initial  goal  of  the  program  was 
to  evaluate  quantitatively  the  impact  of  the  introduction  of  automation  to  the  PDM  line, 
the  reduced  period  of  performance  made  it  impossible  to  directly  measure  this.  The 
largest  impacts  were  a  by-product  of  electronic  dissemination  of  information,  as 
opposed  to  a  reduction  in  the  time  required  to  perform  the  tasks  being  automated.  It 
would  have  required  the  ITI-ALC  tools  to  have  been  fielded  for  several  weeks  in  order  to 
make  measurements  of  these  impacts.  Since  this  was  no  longer  possible,  we 
developed  revised  goals  that  focused  on  user  interface  usability  and  constraints  on 
implementation  of  a  system  similar  to  ITI-ALC  in  the  PDM  environment. 
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1.2  ITI-ALC  Tools 


The  final  ITI-ALC  tools  included  an  E&l  application,  an  Engineering  Assistance  Request 
(202  Form)  application,  electronic  Technical  Order  delivery  system  that  was  integrated 
with  these  applications,  and  an  electronic  identification/signoff  system  that  was  also 
integrated  with  the  two  applications.  The  E&l  application  guided  inspectors  through 
required  inspection  tasks  and  allowed  them  to  record  defects  and  task  completion.  The 
202  Form  application  helped  mechanics  complete  the  202  form,  filling  in  fields  for  them 
or  providing  pull-downs  when  possible.  It  allowed  mechanics  to  record  and  attach 
digital  voice  notes  and/or  user-annotated  digital  photographs  to  the  202  form  prior  to 
submission.  The  integrated  TO  delivery  system  allowed  mechanics  to  view  IPDF 
versions  of  F-15E  technical  data  from  within  either  application,  and  the  electronic 
identification  (ID)  system  was  used  for  logging  on  to  either  application  and  signing  off  of 
tasks  in  the  E&l  application.  The  two  applications  were  hosted  on  a  mobile  computer  (a 
Fujitsu  Stylistic  1200)  which  could  be  worn  by  the  mechanics  during  use. 

The  1200s  were  equipped  with  customized  versions  of  Commercial  Off-The-Shelf 
(COTS)  cases  that  allowed  the  user  to  ’wear1  them  using  a  strap  through  which  the 
user's  neck  and  one  arm  were  inserted.  A  'comfort  pad'  was  added  to  the  lower  edge  of 
the  cases  to  provide  space  between  the  user's  abdomen  and  the  screen  and  to  support 
the  1200  in  a  position  perpendicular  to  the  user's  body.  The  iButton  reader  used  for 
electronic  ID/signofT  was  mounted  just  above  the  right  side  of  the  display,  and  a  ViCam 
digital  camera  was  attached  to  the  right-hand  side  of  the  case.  The  pen  used  for  input 
was  tethered  to  a  ring  on  the  right  (for  right-handed  users)  or  left  (for  left-handed  users) 
side  of  the  case  (see  Fujitsu  1200  in  case  with  accessories  in  Figure  1.2-1). 
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Figure  1.2-1  Fujitsu  Stylistic  1200  with  modified  case 


1 .3  Field  Test  Methodology 

There  were  eleven  final  field  test/Field  Based  Evaluation  #3  (FBE  It 3)  participants. 
Three  of  these  were  E&l  inspectors,  six  were  sheet  metal  mechanics,  and  two  were 
engineers.  Data  collection  methodologies  included  use  of  questionnaires,  evaluator 
observations,  subject  interviews,  and  in  some  cases,  a  paper  walk-through  of  the 
applications.  Each  subject  evaluated  only  the  ITI-ALC  application  that  was  consistent 
with  his  major  job  skill;  i.e. ,  E&l  inspectors  evaluated  only  the  E&l  application,  sheet 
metal  mechanics  evaluated  the  mechanic's  portions  of  the  Engineering  Assistance 
Request  application  and  engineers  evaluated  the  engineer's  portions  of  the  Engineering 
Assistance  Request  application.  All  mechanics  evaluated  the  electronic  technical 
orders  and  electronic  identification  implementation.  Subjects  received  training  that 
varied  in  length  from  one  to  two  hours,  depending  upon  the  speed  with  which  each 
subject  mastered  the  material,  with  two  subjects  being  trained  at  a  time  in  most  cases. 
After  demonstrating  a  minimum  level  of  proficiency  with  the  application,  mechanics  were 
allowed  to  use  the  mobile  systems  on  which  the  applications  were  hosted  to  complete  at 
least  two  tasks  each.  Very  little  assistance  was  provided  to  the  mechanics  during  their 
use  of  the  systems.  Engineers  performed  a  tabletop  discussion  with  evaluators  in  lieu 
of  actual  use  due  to  their  very  different  experience  base. 
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1.4  Field  Test  Results 


The  mechanics  reported  that  the  ITI-ALC  tools  provided  them  with  "increased  access  to 
information",  and  felt  that  novices  could  easily  use  the  tools  with  experience.  They 
reported  that  they  would  use  the  tools  frequently,  and  that  they  would  use  them  in  the 
performance  of  their  primary  job.  They  noted  several  specific  areas  in  which  the  tools 
would  be  helpful,  including: 

•  performance  of  depot  maintenance  on  aircraft 

•  part  number  lookup 

•  guiding  repairs 

•  search  for  general  information 

•  fast,  secure  electronic  173  card  signoffs 

•  faster  and  easier  parts  ordering 

•  increased  access  to  technical  data 

•  aiding  mechanics  in  performing  tasks  outside  their  specialty 

Participants  also  noted  several  areas  in  which  the  tools  require  improvement.  They 
noted  that  the  mobile  system  would  not  be  useful  in  tight  areas  (mechanics  can  barely 
squeeze  into  some  confined  areas  of  the  F-15E  without  the  mobile  computer).  It  is 
reasonable  to  expect  that  users  might  perform  inspections  and/or  repair  work,  then 
record  results  off-aircraft. 

Mechanics  suggested  that  the  screen  used  to  select  electronic  technical  orders  (TOs) 
be  reorganized  to  display  available  TOs  according  to  (1)  aircraft  location,  (2)  technician 
specialty  and/or  (3)  numeric  sequence.  The  tools  currently  display  them  in  either 
alphabetical  order  by  TO  title  or  by  TO  category  (e.g.,  Job  Guides,  Illustrated  Parts 
Breakdowns,  et  cetera).  While  most  of  the  rest  of  the  GUI  seemed  to  be  adequate, 
inspectors  seemed  to  have  difficulty  remembering  to  tap  the  "+"  icon  on  the  'Add 
Defects'  screen  in  order  to  record  the  just-entered  defect. 

The  human/computer  interface  exhibited  other  'rough'  areas.  Users  input  information  to 
the  Fujitsu  1200  by  writing  and  'tapping'  with  a  pen  on  the  display  (tapping  replaces  the 
mouse  button  clicking  used  on  a  desktop  computer).  Almost  every  user  had  difficulty 
'tapping'  icons  with  the  mobile's  pen.  A  very  particular  'tapping  method’  must  be  used 
before  taps  would  be  recognized.  Engineering  analysis  showed  that  this  was  a  problem 
with  Internet  Explorer  4.0,  rather  than  the  Fujitsu  1200.  The  same  screens  worked 
flawlessly  with  Internet  Explorer  3.x,  and  with  Netscape  4.x.  Unfortunately,  this  was 
perceived  by  many  users  as  a  computer  problem. 

Similarly,  once  a  'tap'  was  recognized  by  the  1200,  on  some  screens  the  cursor  would 
flicker  between  the  normal  'pointer'  cursor  and  a  'wait'  cursor  for  several  seconds.  This 
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would  be  followed  by  a  delay  of  up  to  several  seconds  before  any  action  could  be  seen. 
During  this  time,  the  user  was  unsure  if  the  tap  had  registered  or  not.  Engineering 
analysis  showed  that  this,  too,  was  peculiar  to  Internet  Explorer.  The  cursor  flickered 
once  for  every  hidden  field  on  the  screen  (sometimes  over  100  times). 

The  mobile  computers  themselves  proved  to  be  adequate  for  the  task  for  which  they 
were  selected,  although  some  users  expressed  concern  about  the  ruggedness  of  the 
1200s,  saying  that  they  worried  about  breaking  them  during  field  test  use.  Their  120- 
megahertz  (MHz)  clock  speed  provided  reasonable  responsiveness  for  the  thin  client 
implementation.  The  (relatively  expensive)  Thin-Film  Transistor  (TFT)  active  matrix 
display  performed  well  in  the  harsh  lighting  of  the  aircraft  hangar.  The  applications  were 
implemented  as  HTML  pages,  which  were  served  on  demand  to  the  1200s  via  a 
wireless  LAN.  The  wireless  LAN  had  a  peak  advertised  throughput  of  1.6  megabits  per 
second,  which  proved  to  be  adequate  for  small  numbers  of  concurrent  users;  tests  with 
larger  numbers  of  users  were  not  conducted.  The  wireless  LAN  exhibited  some 
sensitivity  to  use  in  close  proximity  (within  3  to  6  feet)  to  operating  microwave  ovens, 
particularly  when  receiving  data. 

Electronic  signoffs  used  a  small  metal  button  ("iButton")  to  identify  mechanics;  the 
button  was  mounted  in  an  angled  plastic  holder  that  could  be  attached  to  a  keychain  or 
worn  around  the  neck  on  a  chain.  During  electronic  identification  operations,  the  user 
inserted  the  iButton  into  the  reader  mounted  above  the  1200's  display,  where  spring 
tension  held  the  iButton  in  place.  While  mechanics  liked  the  speed  and  security  of  the 
iButton-based  signoffs,  they  noted  that  the  button  could  easily  be  dislodged  from  the 
reader  if  left  in  place  during  use  of  the  mobile  system.  This  is  a  serious  concern. 
Aircraft  are  plagued  by  "Foreign  Object  Damage"  (FOD),  which  often  occurs  when  tools 
or  small  parts  are  dropped  into  the  aircraft  and  not  subsequently  recovered.  Failure  to 
properly  secure  iButtons  would  inevitably  result  in  FOD. 

Several  mechanics  had  problems  maintaining  the  1200s  in  an  attitude  suitable  for 
viewing  the  screen  and  providing  input,  particularly  when  taking  digital  photos.  When 
taking  photos,  the  user  viewed  the  digital  camera  output  in  a  window  displayed  on  the 
1200's  screen,  and  turned  the  camera  lens  to  focus.  Once  the  image  was  focused,  the 
user  held  the  camera  still  and  depressed  a  button  on  top  of  the  ViCam  to  ’freeze'  the 
image.  All  users  had  difficulty  focusing  the  camera  adequately,  partly  because  the 
viewing  window  was  too  small  and  partly  because  of  the  relatively  slow  refresh  rate  of 
the  image.  Subsequent  engineering  analysis  showed  that  enlarging  the  screen  window 
results  in  much  better-focused  images,  even  when  the  refresh  rate  is  not  changed.  This 
is  an  important  result,  since  the  ViCam's  refresh  rate  is  inversely  proportional  to  the 
quality  and  detail  of  the  final  image  (i.e.,  better  images  can  be  obtained  at  the  expense 
of  slower  refresh  rates). 

The  users  accepted  the  software  user  interface  design  was  accepted  with  little  comment 
(with  the  exception  of  certain  unlabeled  icons  in  the  E&l  application).  Mechanics  were 
able  to  learn  and  use  the  tools  with  relatively  little  training  (1  to  2  hours),  despite  having 
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little  or  no  prior  computer  experience.  This  is  noteworthy,  as  Air  Force  training  budgets 
are  expected  to  continue  to  dwindle  for  the  foreseeable  future.  We  believe  that  the  user 
interface  design  succeeded  due  to  (1)  minimization  of  the  variety  of  widgets  used  in  the 
applications,  (2)  the  direct  manner  in  which  the  user  interface  design  related  to  the 
existing  work  processes,  (3)  the  extensive  use  of  pictorial  icons,  with  common  images 
incorporated  into  icons  corresponding  to  related  tasks  and  the  availability  of  an  identical 
navigation  bar  on  every  screen  within  each  application. 

1.5  Recommendations 

While  users  generally  felt  that  the  ITI-ALC  tools  would  help  them  to  do  their  jobs  faster 
and  better,  improvements  are  clearly  necessary  in  several  areas.  Those  can  be  broadly 
divided  into  two  major  categories:  hardware  issues  and  user  interface  issues.  The 
following  paragraphs  discuss  our  findings  and  recommendations  in  these  areas. 

Hardware  Issues 

The  Fujitsu  Stylistic  1200s  demonstrated  at  FBE  #3  were  judged  to  be  the  best  fit  for 
mobile  PDM  use  based  on  the  state  of  current  technology.  They  offered  adequate 
performance,  particularly  when  used  as  'thin'  clients.  The  1200's  display  was  bright  and 
clear  even  in  direct  sunlight  and  had  a  wide  viewing  angle,  but  was  relatively  small  (8- 
inch  diagonal)  and  low-resolution  (640x480).  The  small  display  size  resulted  in 
increased  scrolling  by  the  user,  and  limited  the  usefulness  of  the  device  when  viewing 
large  graphics  (e.g.,  schematics,  Illustrated  Parts  Breakdowns).  The  1200  has  I/O 
(input/output)  ports  on  the  body  of  the  unit,  unlike  many  competing  products  that  require 
a  port  expander  for  I/O  port  access.  The  1200  comes  with  a  lithium  ion  battery  pack 
that  provides  for  longer  use  per  charge.  Units  could  probably  be  used  at  least  half  a 
shift  (of  continuous  use)  before  requiring  a  fresh  battery.  Lithium  ion  batteries  offer  a 
longer  battery  pack  life  (i.e.,  more  charge/discharge  cycles  before  battery  must  be 
discarded)  than  other  technologies,  thus  reducing  cost  of  ownership. 

Some  users  were  concerned  that  they  would  damage  the  unit  during  use.  The  1200s 
are  not  ruggedized  to  meet  military  standards,  but  they  are  made  for  mobile  use;  while 
they  would  probably  not  survive  a  10  foot  drop,  incidental  impacts  with  doors,  desks, 
etc.  did  not  seem  to  damage  them.  None  of  the  1200s  used  on  the  program  was 
damaged  despite  heavy  travel,  shipping  and  use  on  the  depot  floor.  When  selecting 
among  otherwise  acceptable  computers  for  PDM  floor  use,  there  are  two  important 
considerations  besides  ruggedness  to  consider.  The  first  is  weight.  A  basic  1200  (right 
out  of  the  box)  weighed  3.9  pounds.  Comparable  (Pentium  100+MHz,  TFT  display) 
ruggedized  machines  weighed  5.8  to  9.6  pounds.  The  1200  as  configured  for  FBE  #3 
(case,  ViCam,  iButton  reader)  weighed  close  to  5  pounds,  and  the  development  team's 
subjective  experience  with  prolonged  use  showed  that  neck  and  shoulder  fatigue  result 
from  use  longer  than  30  minutes.  The  second  consideration  is  cost;  comparable 
ruggedized  units  cost  200+%  the  cost  $5,000  cost  of  the  1200.  The  additional  cost 
could  be  invested  in  spares,  or  reserved  for  future  replacement  of  obsolete  units. 


As  discussed  above,  mechanics  liked  the  iButtons  for  electronic  logon  and  173  card 
signoff,  but  were  concerned  about  their  tendency  to  pop  out  of  the  reader  upon 
relatively  modest  impact  to  the  iButton  holder.  While  not  used  during  FBE  #3,  the 
holders  have  a  hole  for  attaching  to  a  keychain  or  chain  worn  around  the  neck.  Since 
FOD  and  safety  concerns  generally  discourage  mechanics  from  wearing  devices 
around  their  neck,  some  user-acceptable  alternatives  should  be  investigated. 

While  users  did  not  complain  about  throughput  (except  when  searching  the  IPDF  TOs), 
the  developers  believe  that  higher-throughput  RF  LAN  equipment  would  be  required  in 
a  large,  multi-concurrent-user  environment.  This  requirement  has  been  mitigated 
somewhat  by  ITI-ALC's  web  browser-based  applications;  the  LAN  is  used  only  when  the 
user  moves  from  page  to  page,  and  this  should  not  happen  frequently  in  normal  use. 
Use  of  desktop  machines  connected  to  a  wired  LAN  would  further  reduce  the  demand 
for  RF  LAN  bandwidth. 

Summary  of  Hardware  Recommendations 

This  program  stressed  the  state  of  the  art  in  mobile  computing.  Mechanics  cannot 
realistically  be  expected  to  carry  a  device  as  large  and  heavy  as  a  Fujitsu  Stylistic  1200 
with  them  all  day,  but  smaller  platforms  are  not  yet  capable  enough  to  support  the  tasks 
mechanics  must  perform.  The  1200  is  also  too  expensive  for  each  mechanic  to  have  a 
personal  mobile  computer.  As  mobile  computing  becomes  more  widespread,  the  price 
of  highly  capable  machines  will  drop.  The  ideal  platform  for  PDM  use  would  have  a 
larger  screen,  lower  weight,  run  at  least  one  shift  on  a  fully  charged  set  of  batteries  and 
cost  less  than  $500.  A  Windows  CE-based  PalmPC  running  a  CE-compatible  web 
browser  would  meet  most  of  these  criteria,  but  the  screens  of  currently  available  models 
are  not  adequate  for  TO  viewing.  The  iButton  electronic  signature  implementation  will 
work  as-is,  provided  a  reasonable  approach  to  iButton  stowage  can  be  identified. 

User  Interface  Issues 

The  main  problems  with  the  user  interface  related  to  the  use  of  the  pen  for  input.  As 
noted  above,  some  of  these  problems  (tapping,  cursor  jitter)  were  traced  to  Internet 
Explorer  (as  opposed  to  the  Fujitsu  1200  or  the  ITI-ALC  software).  However, 
handwriting  recognition  accuracy  and  ease  of  editing  tool  use  were  still  problematic  at 
the  end  of  the  program.  Use  of  handwriting  editing  tools  was  the  single  most  complex 
task  that  the  mechanics  had  to  master  in  order  to  use  ITI-ALC  applications. 
Consequently,  they  were  trained  to  use  the  pop-up  keyboard  for  editing.  While  the  tools 
available  on  the  keyboard  were  not  as  powerful,  they  were  readily  usable  by  the 
mechanics.  Several  mechanics  eschewed  handwriting  altogether  in  favor  of  the 
keyboard.  It  may  be  that  the  only  reason  more  did  not  make  this  choice  was  their  lack 
of  familiarity  (and  comfort)  with  the  QWERTY  pop-up  keyboard). 

The  Fujitsu  Stylistic  1200  comes  bundled  with  Communication  Intelligence  Corporation's 
(CIC)  Handwriter  recognition  package.  The  newly  available  CalliGrapher  from 
ParaGraph  International  offers  much  better  recognition  accuracy  (including  recognition 
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of  cursive,  which  CIC's  package  does  not  do  at  all)  and  easier  to  use  editing  tools. 
Unfortunately,  this  package  was  not  ready  in  time  for  use  at  FBE  #3,  and  cannot  be  fully 
integrated  with  the  1200  because  no  WinTab  driver  has  been  written  for  the  1200.  The 
developer's  experience  was  that  CIC's  software  could  yield  >95%  accuracy  when  used 
by  an  experienced  operator;  a  preview  version  of  CalliGrapher  yielded  similar  results 
with  no  previous  experience.  Handwriting  input  should  be  a  viable  alternative  in  the 
near  future. 

The  other  major  user  interface  issue  concerns  difficulty  focusing  the  ViCam  when 
creating  an  Engineering  Assistance  Request.  Developers  learned  after  FBE  #3  that 
better  focus  could  be  obtained  through  use  of  a  larger  image  display  window.  However, 
since  the  image  is  updated  at  a  rate  of  about  once  per  second,  the  time  lag  between 
changing  focus  and  seeing  the  resulting  image  on  the  display  still  requires  some 
patience  and  care  on  the  part  of  the  user.  Integration  of  a  good  autofocus  digital 
camera  would  result  in  more  uniformly  focused  images  with  less  user  effort.  Cameras 
of  apparently  adequate  quality  and  reasonable  price  were  just  being  introduced  as  ITI- 
ALC  ended. 

Summary  of  User  Interface  Recommendations 

The  use  of  a  limited  widget  set  and  a  direct  mapping  between  the  user  interface 
implementation  and  tasks  with  which  the  user  was  familiar  helped  bridge  the  gap 
between  the  mechanics'  task  expertise  and  their  lack  of  computer  familiarity.  We  found 
that  the  mechanics  need  to  be  introduced  to  basic  computer  concepts  significantly 
increased  the  amount  of  information  they  had  to  master  during  training.  Consequently, 
they  had  great  difficulty  with  the  relatively  complex  FBE  #2  user  interface  (which  was 
similar  to  that  of  office  automation  tools),  while  enjoying  more  success  with  the 
simplified  FBE  #3  interface. 

This  user  interface  eliminated  as  much  manual  data  input  as  possible.  This  was  done 
by  providing  pre-filled  information  or  user-selectable  pull-downs  containing  a  list  of  valid 
entries  for  a  given  field.  Descriptive  fields  requiring  free-form  user  input  still  pose  a 
problem.  An  improved  handwriting  recognition  package  is  the  best  solution  at  the 
current  time.  Today's  voice  recognition  systems  offer  less  accuracy  than  a  good 
handwriting  package  in  a  quiet  office  environment;  in  the  (sometimes  very)  noisy  depot, 
they  would  have  even  more  difficulty.  No  currently  available  voice  recognition  packages 
do  well  at  both  application  navigation  arid  voice  dictation.  Even  those  intended  for 
navigation  do  not  work  well  with  web-based  applications  because  of  the  non-static 
nature  of  the  application  screens. 
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2.  ITI-ALC  Phase  II  Introduction 


2.1  Background  and  Purpose 

The  ITI-ALC  Phase  II  research  and  development  program  continues  the  work  performed  under 
the  ITI-ALC  Phase  I  effort  (Contract  Number  F41624-94-C-5021)  which  determined  the 
functional  requirements  of  an  information  integration  capability  to  support  Programmed  Depot 
Maintenance  (PDM)  activities.  The  goal  of  this  effort  has  been  to  develop  and  demonstrate 
technology,  which  represents  a  subset  of  the  total  functionality  identified  in  the  Phase  I  System 
Segment  Specification  (SSS).  The  intent  of  the  ITI-ALC  capability  is  to  improve  PDM 
operations  by  making  maintenance  information  more  accessible,  fully  integrated,  and  easy-to- 
use.  The  focus  of  the  ITI-ALC  Phase  II  technology  has  been  to  improve  PDM  activities.  One  of 
the  guiding  Phase  II  objectives  has  been  to  provide  prototypes  that  will  demonstrate  the 
effectiveness  of  as  many  of  the  Phase  I  Business  Process  Improvements  (BPIs)  as  possible.  Table 
2.1-1  maps  the  Phase  I  BPIs  to  the  Phase  II  Prototype. 

Table  2.1-1  Mapping  of  Phase  I  BPIs  to  Phase  II  Prototypes 


Evaluation 

BPI#  BPI  Description  &  inventory 

1  Process  8’  Terminology  Coordination  x 

2  Planning  Process  Enhancement  x 

3  Acquire  Parts  * 

4  Data  Sharing  Among  All  Levels  of  x 


Maintenance 

5  Production  Responsibility  Centers 

6  Component  Parts  Acquisition  Policy 
Changes 

7  Visibility  into  Part  Avail  ability 

8  Electronic  Signatures  x 

9  Performance  Metrics  Based  on  Actual  Data 

10  User  Technical  Information  Presentation  x 

System 

11  Pre-Planned  Over  and  Above/  x 

Unpredictable 

12  Planning  Responsibility  Centers 

13  Automated  and  Integrated  Technical  and  x 

Diagnostics  Information 

14  Multi -skilled  Mechanics 

15  Three  Shifts  of  Effort 


Collaboration 


x 


The  results  of  the  process  re-engineering  activities  performed  in  Phase  I  have  been  expanded  in 
the  design  and  demonstration  of  the  ITI-ALC  Phase  II  capability.  These  ITI-ALC  proof-of- 
concept  demonstrations  have  successfully  been  performed  at  WR-ALC  (WR-ALC)  Air  Logistic 
Center  (ALC).  These  prototypes  have  successfully  demonstrated  the  potential  to  increase 
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technicians’  effectiveness  and  efficiency,  thereby  significantly  reducing  the  flow  days/cost 
needed  to  complete  PDM  activities.  These  ITI-ALC  Phase  II  proof-of-concept  prototypes  were 
implemented  using  an  iterative  user  centered  approach.  This  approach  is  depicted  in  Figure 
2.1-1. 


ITI-ALC 
Phase  I 

AREP 

Results 

Prelim. 

Visits 


PDM  Phases 

1 .  Work  Package 
Negotiations 

2.  Pre-Weapon 
System 
Reception 

3.  Weapon 
System 
Reception 

4.  Pre-Dock 

5.  Mod  Dock 

6.  Post-Dock 

7.  Post-Maintenance 
Data  Analysis 


Potential 
technologies  for 
users  to  examine 

* 

Users’  inputs 

Pilot 

implementations 
(people  use  and 
evaluate) 


Multiple 

Iterations 


/ 


Figure  2.1-1  User  Centered  Program  Approach 


System  subsets  were  taken  to  the  WR-ALC  for  evaluation  by  users  at  multiple  events.  Feedback 
from  each  event  was  incorporated  into  the  next  prototype  development  iteration.  Each 
interaction  with  the  end  users  was  used  to  obtain  detailed  feedback  regarding  the  usefulness  of 
the  chosen  technologies,  as  integrated  into  the  demonstration  prototypes.  The  activities  and 
results  from  each  event  will  be  discussed  in  this  document. 
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2.2  ITI-ALC  Team  Interaction 

The  ITI-ALC  Phase  II  Team  is  comprised  of  the  AFRL/HESR,  Lockheed  Martin  Information 
Systems  (LMIS),  and  Carnegie  Mellon  University  (CMU).  The  WR-ALC  AREP  and  FI 5  PDM 
line  users  have  played  a  critical  role  in  influencing  the  program  direction  and  evaluation  of  the 
ITI-ALC  prototypes.  The  program  structure  and  commitment  to  soliciting  feedback  from  the  end 
user  community  required  frequent  team  interactions.  Figure  2.2-1  presents  a  flow  of  the  work 
products  exchanged  between  the  team  members. 


AREP /Needs  ITI-ALC  Phase  II  Team 


Air  Force 
Research 
Lab 


Stafc/s.  MDPMS 


P/iase  7 


:Ffod(MM\ 


Program  Direction 


LM 

Team 


Program 1  & :  Tech  ft  leaf 
^  Management  ^ 


; ;  :Desigii\  : : 
Work  Products 


Prototypes  .  j| 
Concepts  / 


YA 


Feedback 


End 


Users 


Field  Validation 
PbdSeTWorfc 
Fieldable  Tools 
fbiriPd^ 


Figure  2.2-1  Team  Interaction 


u 


The  LMIS  Demonstration  and  Development  (DD)  and  Demonstration  Evaluation  (DE) 
Integrated  Product  Team  (IPT)  Leads  facilitated  the  flow  of  communication  and  work  products 
between  the  product  team  members,  as  detailed  in  Figure  2.2-2. 


2.3  Study  Site  and  Personnel 

The  WR-ALC  F-15  PDM  line  was  selected  as  the  primary  site  for  prototype  evaluation  and 
demonstration  events.  The  ITI-ALC  program  received  outstanding  cooperation  from  the  WR- 
ALC  AREP  community,  F-15  PDM  line  supervisor,  and  F-15  PDM  line  personnel  (inspectors, 
mechanics,  engineers,  planners,  and  schedulers)  to  accomplish  the  program  goals.  Refer  to 
section  3.4  for  more  detailed  information  on  all  study  event  participants. 


2.4  Schedule  of  Study  Events 


There  were  seven  primary  evaluation/demonstration  events  conducted  at  WR-ALC  and  one  data 
gathering  event  conducted  at  Tinker  Air  Force  base  between  June  1997  and  October  1998. 
Field-based  Evaluation  #1  (FBE  #1)  focused  on  informing  the  F-15  program  about  the  ITI-ALC 
Phase  II  plans  and  promising  technologies.  At  this  event  the  team  initiated  our  user-centered 
approach  to  refine  and  update  requirements  that  had  changed  since  ITI-ALC  Phase  I  (which 
ended  in  1995).  Two  data  collection  visits  to  WR-ALC  and  one  visit  to  Tinker  AFB  followed 
this  formal  event.  Each  of  these  research  field  visits  involved  extensive  interviewing  and 
observational  activities.  Based  on  the  four  initial  events,  prototype  systems  to  support  two 
applications  were  developed  at  Lockheed  Martin  and  Camegie-Mellon  University.  Early 
versions  of  these  prototypes  were  taken  to  Robins  in  November,  1 997  for  limited  user  testing  and 
feedback  about  functionality,  form  factors  of  hardware,  etc.  All  of  this  information  was 
incorporated  in  the  next  iteration  of  the  prototypes.  A  second  FBE  was  held  at  WR-ALC  in 
February  and  March  1998.  This  event  was  followed  by  an  Air  Force  “shakedown”  test  of  the 
prototypes  in  July.  The  final  user  evaluation  of  the  two  prototypes  was  held  at  Robins  AFB  in 
October  1998.  A  detailed  description  of  these  events  is  provided  in  Section  3  of  this  report. 


2.5  Technician  and  Other  Participants'  Activities 

The  WR-ALC  ALC  F-15  PDM  technicians  have  been  the  primary  end-user  focus  of  the  ITI-ALC 
Phase  II  prototypes.  Feedback  on  functionality  and  usability  was  also  solicited  from  supervisors, 
planners,  schedulers,  and  engineers  for  the  applicable  PDM  support  role  that  each  individual 
performs.  A  user  group  of  twenty  individuals  (4  inspectors,  6  skin  mechanics,  2  Aircraft  (A/C) 
mechanics,  2  A/C  electrical  technicians,  2  supervisors,  1  scheduler,  and  1  engineer)  worked  with 
the  ITI-ALC  team  for  the  duration  of  the  project.  They  participated  in  most  of  the  seven  events 
mentioned  in  section  2.4,  especially  in  updating  the  requirements,  trying  out  prototypes  over 
time,  and  providing  their  candid  feedback  to  the  evaluators  during  each  prototype  test. 
Additionally,  3  structural  engineers,  2  planners,  2  AREP  representatives,  and  1  Non-Destructive 
Inspection  (NDI)  expert  were  involved  in  the  requirements  updating  and  in  providing  inputs 
about  the  prototype  development  as  it  progressed.  A  more  detailed  description  of  participants’ 
activities  is  provided  in  section  3  of  this  document. 
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2.7  Organization  of  this  Report 

This  document  has  been  organized  as  requested  by  the  ITI-ALC  Statement  of  Work  (SOW)  to  be 
consistent  with  ANSI/NISO  Standard  Z39.18-1995,  Scientific  and  Technical  Reports  -  Elements, 
Organization,  and  Design.  The  outline  as  implemented  in  this  document  is  as  follows: 

Section  1,  ITI-ALC  Phase  II  Summary,  summarizes  the  program  approach,  key  events  of  the 
program,  event  results,  conclusions,  recommendations. 

Section  2,  ITI-ALC  Phase  II  Introduction,  provides  more  detail  on  the  ITI-ALC  background, 
purpose,  study  site,  study  personnel,  study  events 

Section  3,  Methods  and  Procedures,  contains  analytic  information  about  the  types  of  measures 
employed  at  each  of  the  significant  user  interaction  events. 

Section  4,  Results,  contains  analytic  information  about  the  types  of  measures  employed  at  each 
of  the  significant  user  interaction  events. 

Section  5,  Conclusions,  summarizes  the  results  of  the  significant  user  interaction  events. 

Section  6,  References,  lists  document  references  made  in  this  report. 

Section  7,  Acronyms  and  Abbreviations,  spells  out  the  acronyms  and  abbreviations  used  within 
this  document. 

The  appendices  of  this  report  contain  information  on  the  Trade  studies  performed  for  this 
program,  questionnaires,  and  segments  of  collected  user  feedback. 
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3.  Methods  and  Procedures 


The  following  sections  describe  the  prototype  design  and  development,  the  schedule  of  major 
study  events,  the  maintenance  tasks  performed  at  those  events  (as  applicable),  the  personnel 
involved  in  each  event,  and  the  site  logistics  support  provided  to  conduct  each  event.  The 
methods  for  participant  training  and  the  data  collection  procedures  are  also  presented,  as 
applicable. 

3.1  Prototype  Design  &  Development 

The  process  for  the  design  and  development  of  the  Phase  II  prototypes  was  initiated  with  a 
review  of  the  Phase  I  work  products  identified  in  the  ITI-ALC  SOW.  Multiple  trips  were  then 
conducted  to  talk  with  the  end  users  and  gather  feedback,  as  described  in  sections  3.2  and  beyond 
of  this  document.  This  interaction  allowed  the  ITI-ALC  team  to  understand  the  depot 
environment  changes  initiated  by  the  Re-engineering  Office  since  completion  of  Phase  I.  It  also 
allowed  the  ITI-ALC  team  to  focus  on  those  areas  that  targeted  BPIs  of  particular  concern  and 
could  contribute  to  the  re-engineering  effort  without  duplicating  current  initiatives.  The  initial 
Phase  II  software  requirements  were  drafted  as  a  result  of  these  actions,  resulting  in  the 
Computer  Software  Components  (CSCs)  defined  in  Figure  3.1-1. 


Figure  3. 1-1.  ITI-ALC  Phase  II  CSC  Decomposition 


As  the  program  matured  and  additional  information  became  available  on  re-engineering 
activities,  the  prototypes  for  the  planning,  scheduling,  routing  and  tracking  areas  were  down 
scoped  from  the  Phase  II  effort.  When  the  program  was  shortened  from  36  to  19  months,  they 
were  dropped  altogether.  The  following  sections  will  present  the  system  architecture  and  design 
for  the  Collaboration,  Tech  Data  Presentation,  Inspection,  and  Electronic  Identification  CSCs. 
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3.1.2  ITI-ALC  System  Architecture 

The  software  and  system  architecture  is  presented  in  Figures  3. 1.2-1  and  3. 1.2-2. 


Windows  NT  Server, 
Oracle  7.3, 

Internet  Information  Server. 
Proxfm  RF  LAN  Driver, 
Java  Servlets  (DHTML) 
Mail  Server 


Fujitsu 

Windows  95.  Internet  Explorer  4.0  $P  1 . 
Proxim  RF  LAN  Client  Driver. 
Handwriting  Recognition  BAAL 
Adobe  Acrobat  PDF  Viewer 


Engineer's  Desktop 


Windows  95.  Internet 
Explorer  4.0  SP1 . 
Adobe  Acrobat  PDF  Viewer 


Figure  3.1. 2-1.  ITI-ALC  Software  Architecture 


Figure  3. 1.2-2.  ITI-ALC  System  Architecture 


Selection  and  implementation  of  a  mobile  computing  platform  suitable  for  the  depot  environment 
was  a  result  of  this  evolutionary  user  centered  effort.  Three  different  mobile  client  platforms 
were  implemented  over  the  iterative  life  cycle  of  ITI-ALC  Phase  II.  The  Apple  Newton,  Fujitsu 
Point  510,  and  Fujitsu  1200  are  shown  in  Figures  3. 1.2-3  to  3. 1.2-5. 
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Figure  3. 1.2-4.  Fujitsu  Point  510  Client  Platform 
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Figure  3. 1.2-5.  Fujitsu  1200  Client  Platform 


18 


3.1.3.  Collaboration  Computer  Software  Component  (CSC)  Design 

The  Collaboration  CSC  design  was  implemented  as  presented  in  Figure  3. 1.3-1. 


Figure  3.1. 3-1.  Collaboration  CSC  Architecture 


The  collaboration  tool  allows  a  mechanic  to  electronically  submit  an  AFMC  Form  202  (request 
for  Engineering  Assistance).  This  collaboration  tool  makes  the  AFMC  Form  202  available  to 
engineer  within  minutes,  opposed  to  several  days  with  the  current  process.  The  collaboration 
tool  provides  a  method  for  a  mechanic  to  create  an  engineering  request  form  modeled  after  an 
AFMC  Form  202.  The  form  captures  the  mechanic  identification,  aircraft  tail  number,  date  and 
time,  and  a  problem  description.  The  collaboration  tool  provides  a  method  for  a  mechanic  to 
take  a  digital  picture  of  the  problem,  annotate  the  picture,  and  include  this  digital  picture  with  the 
engineering  request  form.  The  mechanic  may  also  include  a  sound  clip  (a.k.a.  Voice  Note)  or 
saved  sketch  with  the  Form  202. 

The  collaboration  tool  alerts  (Figure  3. 1.3-2)  an  engineer  via  e-mail  to  the  existence  of  an  open 
engineering  request  form.  The  engineer  may  then  view  and  respond  to  the  engineering  request 
form  from  their  web  browser. 
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Figure  3. 1.3-2.  ITI-ALC  Collaboration  Tool  Entry  Screen 


3.1.4  Technical  Data  Presentation  CSC 


The  Technical  Data  Presentation  CSC  design  was  implemented  as  presented  in  Figure  3. 1.4-1. 


Client  (Mechanic) 


Server 


Client  (Engineer) 


Figure  3. 1.4-1.  Technical  Data  Presentation  CSC  Architecture 


The  technical  data  presentation  tool  enables  a  user  to  electronically  navigate  and  display 
technical  manuals.  The  user  (mechanic  or  engineer)  is  able  to  view  electronic  job  guides,  fault 
isolation  manuals,  and  schematic  and  wiring  diagrams.  The  tool  supports  multi-page  technical 
manual  diagrams.  The  tool  allows  the  user  to  zoom  in  or  zoom  out  of  a  diagram  in  order  to  place 
the  currently  displayed  diagram  section  in  context.  The  miniaturized  ('zoomed  in')  view  allows 
the  technician  to  easily  navigate  from  one  diagram  section  to  another.  The  technical  data  tool 
allows  a  user  to  select  manuals  by  document  title  or  category.  The  tool  supports  the  ability  to 
receive  and  display  technical  manuals  from  a  server  one  page  at  a  time. 

The  technical  data  tool’s  presentation  format  (Figure  3. 1.4-2)  is  based  on  COTS  World  Wide 
Web  document  browsing  technology  (Internet  Explorer  4.0)  and  COTS  IPDF  viewer  plugin 
(Adobe  Acrobat  Reader). 
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3.1.5  Inspection  CSC 

The  Inspection  CSC  design  was  implemented  as  presented  in  Figure  3. 1.5-1. 


Client  (Planner/ 


Figure  3. 1.5-1.  Inspection  CSC  Architecture 


The  inspection  tool  allows  a  technician  to  record  defects  and  the  areas  of  the  aircraft  on  which 
the  defects  were  found.  The  inspection  tool  implements  the  electronic  ID  tool  to  authenticate  the 
user,  both  for  security  reasons  and  for  the  ability  to  assist  in  tracking  and  scheduling  maintenance 
procedures.  The  tool  provides  a  means  to  uniquely  record  which  aircraft  (and  aircraft  region)  is 
being  inspected.  The  inspection  tool  displays  defects  and  options  that  are  applicable  to  each 
area.  The  inspection  tool  has  been  populated  with  the  most  common  aircraft  defects  (as 
identified  by  inspectors  at  the  test  site).  These  defects  are  available  as  selectable  options.  When 
a  technician  encounters  a  defect  that  is  not  already  an  option,  the  inspection  tool  provides  them 
with  a  method  for  adding  an  ad-hoc  defect.  This  implementation  discourages  the  widespread 
creation  and  use  of  ad-hoc  defects,  a  detriment  to  later  defect  analysis.  The  mobile  inspection 
tool  client  platform  (Fujitsu  1200  Tablet  computer)  has  been  chosen  (and  tailored)  to  allow  the 
inspector  to  take  it  directly  to  the  aircraft  area  that  is  being  inspected.  The  inspection  tool 
records  technician-identified  defects  of  each  area  in  a  repository  (Oracle  database).  This 
repository  format  allows  for  the  potential  to  export  the  collected  data  to  the  scheduling  system, 
allowing  the  repair  of  defects  to  be  planned  and  scheduled.  As  an  interim  step,  the  repository 
will  be  used  to  a  display  the  identified  discrepancies  and  completed  173  inspection  steps. 

The  inspection  tool’s  display  format  (Figure  3. 1.5-2)  is  based  on  COTS  World  Wide  Web 
document  browsing  technology  (Microsoft  Internet  Explorer  4.0SP1). 
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Figure  3. 1.5-2.  ITI-ALC  Inspection  Tool  Entry  Screen 


3.1.6  Electronic  Identification  CSC 


The  electronic  identification  tool  (Figure  3. 1.6-1)  provides  a  means  for  an  individual  to  be  uniquely 
identified  to  each  of  the  previously  described  mobile  client  prototype  tools.  This  identity  is  then  used  to 
tailor  the  each  tool  to  the  individual.  For  example,  if  an  individual  is  an  inspector,  only  the  inspectors’ 
173  inspection  tasks  and  defects  are  presented.  If  the  individual  is  a  sheet  metal  mechanic,  the  option  to 
submit  a  request  for  engineering  assistance  (Form  202A)  is  presented.  The  electronic  identification  tool 
has  been  implemented  using  the  iButton  technology.  This  tool  also  provides  a  paperless  means  of 
'stamping  off  on  the  completion  of  1 73  inspection  tasks,  which  is  similar  to  the  currently  implemented 
physical  'stamp'  methodology.  The  tool  typically  completes  determination  of  the  identification  of  an 
individual  within  5  seconds.  Because  each  electronic  iButton  has  a  unique  identifier,  the  tool  will 
accurately  identify  the  individual  1 00%  of  the  time,  and  is  not  physically  'intrusive'  to  the  individual. 
The  chosen  tool  is  capable  of  operating  in  the  hangar  environment,  as  the  typical  temperature,  humidity, 
lighting,  and  ambient  noise  in  that  environment  will  have  no  effect  on  the  identification  of  each  physical 
iButton. 


Please  Insert  Your  Electronic  Stamp  Into  The  Receptor. 


Figure  3. 1.6-1.  ITI-ALC  Client  Logon  Screen 
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3.2  Supporting  Trade  Studies 


The  trade  studies  performed  under  the  ITI-ALC  contract  are  provided  in  Appendix  A.  They  capture 
quantitative  and  qualitative  data  about  the  different  products  researched  by  the  ITI-ALC  team.  The  trade 
studies  include  a  summation  of  the  trade  identifying  the  best  candidate,  any  metrics  or  test  data  collected 
during  the  trade,  a  trade  study  assessment  table  for  each  product  or  tool  being  studied,  experience  with 
the  products,  and  product  data  sheets  (marketing  information)  from  the  vendors. 

3.3  Description  of  Maintenance  Tasks 

During  FBE  #1  the  user  group  indicated  that  they  expected  ITI-ALC  Phase  II  resources  would  be  used 
most  productively  on  developing  prototypes  to  support  their  work  in  three  areas.  The  three  depot 
maintenance  tasks  identified  were: 

(1)  Recording  and  distributing  Evaluation  and  Inventory  Inspection  information. 

(2)  Documenting  and  communicating/collaborating  between  skin  mechanics  and  engineers  about 
situations  requiring  engineering  assistance  (Over  and  Aboves),  especially  for  vertical  tail  assemblies 
on  the  F-15  aircraft. 

(3)  Enabling  effective  collaboration  among  technicians  conducting  operational  checkouts  on  an  aircraft. 

The  ITI-ALC  team  documented  the  “as  is”  current  situation  for  each  of  these  three  application  areas 
immediately  after  FBE  #1.  Two  of  the  three  areas  were  selected,  demonstrated,  and  evaluated  using 
these  depot  maintenance  tasks: 

(1)  Evaluation  and  Inventory  (Inspection) 

(2)  Collaboration  about  “Over  and  Aboves”  for  the  Vertical  Tail  Assembly  of  the  F-15. 

The  third  area  was  not  pursued,  due  to  the  heavy  dependence  on  electronic  technical  data,  especially 
schematic  drawings,  which  proved  to  be  problematic  to  acquire  for  ITI-ALC  Phase  II  use. 

Visionary  scenarios  for  each  of  the  two  application  areas  were  developed  and  reviewed  by  the  user 
group.  These  scenarios  served  as  the  basis  for  the  development,  evaluation  and  measurement  efforts  for 
ITI-ALC  Phase  II  as  they  provided  a  clear  indication  of  the  functionality  and  performance  support 
capabilities  users  would  expect  from  the  technologies  used  in  each  prototype  system.  Each  of  the  tasks 
and  the  usability  evaluation  efforts  associated  with  the  ITI-ALC  Phase  II  development  efforts  are 
described  below. 
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3.3.1  Evaluation  and  Inventory  (E&l)  Task  Description 

This  study  evaluated  the  impact  of  the  E&I  prototype  on  aircraft  inspections  being  conducted  when  the 
aircraft  reaches  the  Mod-Dock  phase  of  the  PDM  cycle.  The  complete  evaluation  and  inventory 
inspection  task  involves  ninety-three  1 73  cards,  and  is  expected  to  be  completed  within  a  ten-day  period. 
The  individuals  who  perform  this  inspection  walk  around  the  aircraft,  and  use  ladders  to  climb  into  the 
cockpit,  or  to  view  the  fuselage  above  eye-level.  They  visually  and  tactilely  examine  each  plane  section 
by  section.  The  inspection  tasks  differ  in  terms  of  length  and  complexity,  and  are  almost  always  a  single 
person  effort.  However,  the  current  F-15  inspection  arrangement  at  Robins  AFB  involves  a  two-man 
team  comprised  of  a  very  experienced  inspector  and  a  novice  inspector.  There  appear  to  be  personal 
differences  in  the  approach/process  that  inspectors  use  in  conducting  this  work.  Also,  the  order  and 
sequence  of  aircraft  sections  they  work  on  varies  from  one  aircraft  to  another  and  from  one  inspector  to 
another.  Overall,  the  major  time  associated  with  this  task  has  been  in  entering,  verifying  and 
communicating  the  inspection  findings  to  generate  work  for  others.  That  is,  the  inspection  findings 
result  in  parts  ordering  and  scheduling  of  work  tasks  for  appropriate  mechanics. 


3.3.1. 1  Scope  of  E&l  Task 

The  ITI-ALC  Phase  II  E&I  inspection  tool  was  designed  to  enable  inspectors  to  easily  observe  and 
record  defects  from  any  of  the  15  regions  they  normally  consider  on  the  F-15  aircraft.  After  careful 
consideration  of  several  factors,  including  safety  and  task  complexity,  two  areas  of  the  aircraft  were 
selected  for  testing  the  usability  of  the  prototype  during  FBE  #2,  the  shakedown  exercise  and  the  final 
user  evaluation.  The  regions  the  E&I  inspectors  used  were  (1)  the  cockpit  (areas  2  and  2A  of  the 
aircraft)  and  (2)  the  engine  compartments  (areas  12  left  and  right  of  the  aircraft).  These  aircraft  areas 
were  selected  in  consultation  with  first  line  supervisors  and  inspectors  at  WR-ALC.  (Section  9,  Aft 
Fuselage,  was  not  considered  since  this  area  of  the  aircraft  was  involved  in  the  FBE  #2  Collaboration 
prototype  demonstration  and  evaluation.)  Table  3. 3.1. 1-1  shows  the  regions  of  the  aircraft  identified  for 
use  in  the  E&I  inspection  prototype  efforts. 
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Table  3.3.1. 1-1.  FI 5  Aircraft  Inspection  Regions 


Region 

Description 

01 

Forward  Fuselage,  Radome  and  Speed  Brake 

1A 

Nose  Gear,  Well  and  Doors 

02 

Forward  Cockpit 

03 

Left  Engine  Intake  Duct 

04 

Right  Engine  Intake  Duct 

05 

Left  Wing 

06 

Right  Wing 

07 

Center  Fuselage  and  Fuel  Cell 

7A 

Left  Main  Gear,  Well  and  Doors 

7B 

Right  Main  Gear,  Well  and  Doors 

09 

AFT  Fuselage 

10 

Left  Engine 

11 

Right  Engine 

12 

Engine  Compartments 

13 

Aircraft  General 

The  users  evaluated  the  prototype  system  to  determine  its  acceptability  and  its  potential  impact  on 
reducing  inspectors’  effort  associated  with  inspecting  and  recording  E&I  data.  The  version  of  this  tool 
developed  for  the  final  user  evaluation  exercise  enabled  measurement  of  the  impact  of  transmitting  the 
E&I  data  to  those  who  use  it,  e.g.,  planners  and  schedulers.  That  is,  the  prototype  supports  export  of  the 
E&I  inspection  data  for  integration  with  WR-ALC  legacy  systems  (in  particular,  Programmed  Depot 
Maintenance  Scheduling  System  (PDMSS)).  A  task  overview  is  presented  next  to  provide  the  reader 
with  a  sound  understanding  of  the  activities  involved  in  E&I  inspection. 
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3.3.1. 2  E&l  Task  Overview 


For  the  E&I  task,  an  inspector  conducts  a  visual  examination  of  the  aircraft  exterior  (skin,  canopy,  etc.) 
and  records  the  status  of  the  aircraft  panel  or  component.  Also,  aircraft  parts  are  inventoried  so  that 
missing/wom  parts  information  could  be  passed  to  the  Forward  Logistics  Specialist  (FLS)  to 
order/obtain  parts  to  expedite  the  aircraft’s  progress  through  the  Mod-Dock  phase.  The  inspector 
selected  a  section  of  the  aircraft  to  evaluate  (in  the  case  of  FBE  #2,  the  researchers  selected  the  sections 
as  noted  above).  Inspectors'  tools  are  comprised  of  a  flashlight  and/or  mirrors  they  may  use  while  doing 
the  tasks.  For  the  baseline  (without  technology)  condition,  they  had  available  (but  did  not  use)  a  paper 
checklist  of  frequently  found  defects  (Figure  3. 3. 1.2-1)  with  data  fields  to  record:  the  aircraft  area,  the 
station,  the  skill  required  to  do  the  repair,  and  a  brief  description  of  the  discrepancy.  Inspectors  went  to 
the  section  of  the  aircraft  to  be  inspected,  performed  the  inspection,  and  logged  it  in  their  workbook 
(Figure  3. 3. 1.2-2).  They  later  recorded  their  work  by  stamping  the  appropriate  173  work  cards  (Figure 
3.3. 1.2-3). 
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Figure  3.3. 1.2-1  Sample  ofE&l  Frequent  Defect  List 
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Figure  3. 3. 1.2-2  Sample  of  E&l  Workbook  Entry 


Figure  3.3. 1.2-3  Sample  of  E&l  173  Inspection  Work  Card 


These  inspectors  did  not  record  responses  to  each  possible  defect  item  on  the  checklist.  When 
new/inffequent  defects  were  seen,  they  wrote  a  brief  description  of  such  defects  on  a  new  173  card,  and 
indicated  that  these  would  be  manually  entered  later  into  the  U-book  by  the  scheduler  (Figure  3. 3. 1.2-4). 
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Note  that  inspection  of  a  given  section/panel  of  the  aircraft  varies  greatly  with  respect  to  the  time  to 
complete  each  task  (5  minutes  to  an  hour). 
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Figure  3.3. 1.2-4  Sample  ofE&l  Defect  Worksheet  Entry 


3.3.1. 3  Defects  Used  for  the  E&l  Task 

No  physical  defects  were  inserted  for  any  of  the  events.  For  both  the  baseline  and  with  technology 
cases,  the  selected  aircraft  had  not  been  inspected.  The  aircraft  was  selected  in  consultation  with  the 
aircraft  supervisor  who  anticipated  that  real  defects  would  be  present.  No  additional  defects  were 
needed  (or  inserted)  by  test  personnel  for  this  evaluation.  Defects  present  (and  found  by  the  inspectors) 
during  FBE  #2  were  small  cracks  in  the  engine  compartment,  sections  12  left  and  right  and  some 
corrosion  in  the  cockpit,  sections  2  and  2A. 
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3.3.2  Collaboration  (Engineering  Assistance)  Task  Description 

This  task  involves  collaboration  over  time  and  distance  between  two  or  more  individuals.  The  activities 
of  skin  mechanics  who  are  seeking  engineering  assistance  to  determine  a  course  of  action  for  handling 
“Over  and  Above”  defects  should  be  supported  by  the  introduction  of  an  ITI-ALC  Phase  II  prototype. 
These  activities  include  prompt  sending  and  receiving  of  textual  information,  sketches  and  measurement 
information  as  well  as  references  to  Technical  Orders  (TOs).  The  specific  task  used  in  FBE  #2,  the 
shakedown  exercises,  and  the  final  evaluation  was  the  resolution  of  defects  fotind  on  the  vertical  tail 
assembly  of  an  F-15C  aircraft. 


3.3.2. 1  Scope  of  Collaboration  Task 

To  determine  the  scope  of  this  task,  the  ITI-ALC  Phase  II  team  conferred  with  the  supervising  structural 
engineer  to  determine  which  areas  of  the  F-15  aircraft  structure  have  the  most  frequent  occurrence  of 
“Over  and  Above”  defects.  Also,  the  F-15  first  line  supervisor  and  several  skin  mechanics  were 
consulted  about  the  area  of  the  aircraft  to  select  for  the  field-based  evaluation  activities.  The  vertical  tail 
assembly  of  this  aircraft  was  identified  as  the  appropriate  area  to  use  in  the  Collaboration  (Engineering 
Assistance)  prototype.  According  to  historical  data  available  from  the  supervising  structural  engineer, 
on  average,  thirty  requests  for  engineering  assistance  are  generated  each  month  for  this  section  of  the 
aircraft.  The  Non-Destructive  Inspection  (NDI)  engineer  consulted  during  the  ITI-ALC  Phase  II  effort 
also  endorsed  this  selection  of  aircraft  area.  The  following  Figure  3. 3. 2. 1-1  shows  this  section  of  the 
aircraft. 
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Figure  3. 3. 2. 1-1  Vertical  Tail  Assembly  for  Collaboration  Prototype 

Evaluation  of  the  Collaboration  prototype  was  aimed  at  gaining  feedback  from  actual  users  about  the 
acceptability  and  potential  impact  of  efficiency  gains  they  could  expect  to  realize  using  such  a  prototype. 

A  task  overview  is  provided  next  to  insure  that  the  reader  has  a  detailed  understanding  of  the  tasks  the 
ITI-ALC  prototype  supports  for  skin  mechanics  and  structural  engineers. 


Z.Z.2.2  Collaboration  Task  Overview 

During  the  disassembly  phase  of  the  planned  depot  maintenance  process,  a  skin  mechanic  notices  that 
there  is  an  apparent  crack  in  the  vertical  tail  assembly  of  the  plane  and  believes  that  it  needs  to  be 
repaired.  Because  this  repair  is  not  in  the  Technical  Orders,  the  mechanic  must  request  direction  and 
approval  to  do  this  repair.  He  obtains  and  completes  Part  A  of  a  Form  202  (Figure  3. 3. 2.2-1), 
Engineering  Assistance  Request,  describing  the  location  and  nature  of  the  defect  -  often  including  a 
rough,  hand-done  sketch.  A  scheduler  and  a  planner  log  the  form.  They  (along  with  a  first-line 
supervisor)  decide  whether  the  form  should  be  sent  to  Engineering.  If  the  aircraft  schedule  will  be 
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seriously  impacted  by  this  repair  activity,  then  the  form  may  be  faxed  or  hand  carried  to  Engineering 
(about  4  long  blocks  away  from  the  F-15  hanger).  When  it  is  sent,  the  structural  engineering  supervisor 
assigns  the  Form  202  to  an  engineer  who  visits  the  hanger  to  inspect  the  defect,  calls  the  technician  to 
ask  about  the  severity  of  the  defect,  or  simply  determines  from  the  Form  that  a  repair  is  required. 
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The  engineer  completes  Part  B  of  the  form  (Figure  3.3.2.2-1)  and  specifies  the  appropriate  repair, 
including  drawings  of  the  approved  fix.  This  information  is  either  hand  delivered  or  faxed  to  the 
technician  in  the  hanger.  The  technician  executes  the  repair,  but  often  needs  to  clarify  what  the  engineer 
advises  by  calling  or  requesting  a  visit  to  the  hanger  from  the  engineer.  The  technician  works  from  a 
platform  ladder  about  15  feet  above  the  hanger  floor,  at  the  tail  of  the  plane.  The  elapsed  time 
associated  with  the  Form  202  process  for  vertical  tail  assembly  defects  has  been,  on  average,  seven  to 
ten  days. 


3.3.2.3  Defects  Used  for  the  Collaboration  Task 

An  aircraft  was  selected  in  consultation  with  the  aircraft  supervisor  from  the  F-15  line  for  FBE  #2,  for 
the  shakedown  exercise  and  for  the  final  evaluation.  There  were  defects  identified  by  Non-Destructive 
Inspection  (NDI)  group,  but  not  yet  seen  by  the  skin  mechanics  participating  in  the  prototype  usability 
testing.  These  defects  included  corrosion  and  cracks.  No  additional  physical  defects  were  needed  (or 
inserted)  by  test  personnel  for  these  evaluations. 


3.4  Study  Participants 

The  participants  that  contributed  to  this  study  consist  of  individuals  from  AFRL/HESR,  Lockheed 
Martin,  Carnegie  Mellon  University,  PDM  flight  line  supervisors,  planners,  schedulers,  inspectors, 
mechanics,  and  engineers  that  make  up  the  DE  and  DD  IPTs.  DE  IPT  teams  conducted  most  user- 
centered  events.  Each  team  would  be  comprised  of  an  interviewer/observer/researcher  from  Carnegie 
Mellon  University  (CMU)  and  a  videographer.  In  the  cases  in  which  training  was  administered,  each 
team  would  use  a  prepared  script  to  insure  consistency  in  training.  Additional  Demonstration 
Development  (DD)  IPT  personnel  provided  support  for  network  and  platform  performance  issues  at 
selected  events.  The  interaction  of  the  additional  personnel  with  study  participants  was  minimal. 

3.5  Data  Collection  Procedures 

At  the  inception  of  the  Field-based  Evaluation  efforts,  each  participant  completed  a  non-disclosure  form 
and  an  initial  questionnaire  containing  demographic  and  background  information  for  use  in  the  study. 
Then  interview  data  were  collected  via  audio  and  video  recording,  and  researchers’  observations.  Also, 
substantial  archival  data  were  collected,  e.g.,  blank  and  completed  paper  forms,  schedules  of  aircraft 
movement  through  PDM,  spread  sheets  showing  current  processes,  etc.  During  FBE  #2,  the  shakedown 
exercise  and  the  final  evaluation  effort,  additional  data  collection  efforts  occurred.  These  are  described 
in  the  sub-sections  to  follow. 
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3.5.1  Usability  Study  Events 

In  February  and  March  1 998,  "without  technology"  (baseline)  and  with  technology  trials  of  the  ITI-ALC 
Phase  II  prototypes  were  conducted.  The  main  purposes  of  these  trials  were:  (1)  to  determine 
modifications  required  in  the  iterative  development  process  and  (2)  to  begin  to  collect  data  about  the 
potential  impact  of  these  prototypes  on  mechanics’  and  engineers’  work  processes  and  task  performance. 

Four  major  sources  of  data  were  drawn  upon  during  FBE  #2: 

•  Video/audio  taping  of  actual  behavior, 

•  Automated  data  logs  of  user  behavior  (client  and  server  time  stamped  histories  of  most  interactions) 

•  Survey/interviews  eliciting  users’  subjective  reactions  to  ITI-ALC  prototypes,  and 

•  Observational  evaluation:  watching  and  monitoring  users’  performance  of  specific  tasks  (enacting 
each  scenario)  with  the  ITI-ALC  prototypes. 

Specific  data  collection  was  done  primarily  for  usability  feedback  to  the  developers  and  included  efforts 
to  measure: 

•  time  for  sub-task  and  task  completion  (e.g.,  finding  information  needed,  completing  inspection  of  an 
area,  marking  a  defect), 

•  quality  of  task  performance  (is  the  inspection  performed  thoroughly  etc.), 

•  amount  and  usability  of  information  available  to  maintenance  personnel, 

•  the  flow  of  information  (do  people  follow  their  process  or  change  it  when  using  the  prototypes), 

•  the  amount  and  nature  of  assistance  needed  to  use  the  prototype(s), 

•  the  ease  of  movement  (mobility)  of  users  with  prototypes,  especially  in  confined  or  hard  to 
reach/maneuver-in  spaces,  and  the  ease  of  use  of  prototypes  in  variable  lighting,  temperature,  noise 
and  other  conditions  on  the  PDM  line. 

For  the  survey  effort,  a  post-questionnaire  (Appendix  B)  was  administered  to  elicit  inspectors’  or  skin 
mechanics’  reactions  immediately  following  use  of  the  prototype  systems.  This  instrument  included 
both  open  and  closed  questions.  This  was  also  supplemented  with  written  questionnaires  with  limited, 
structured  interviews  that  were  audiotaped  for  transcription  to  insure  accurate  analysis  of  verbal 
feedback. 

For  the  observational  effort,  rigorous  video  recording  of  training  and  use  sessions  was  conducted  to 
capture  detailed,  time-stamped  records  of  technicians’  performance  during  task  execution.  These  data 
were  correlated  with  the  automatically  collected  data  to  show  how  the  prototype  was  used  at  any  given 
moment.  Additionally,  verbal  protocols  (think-alouds)  were  elicited  to  capture  each  technician's  spoken 
observations  and  thoughts  as  they  used  the  prototype. 
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3.5.1. 1  Usability  Procedures 


The  following  procedures  were  employed  to  evaluate  usability  of  each  of  the  prototypes.  During  the 
first  (baseline)  session,  technicians  were  briefed  about  the  overall  evaluation  process  and  those  who 
were  not  involved  in  FBE  #1  filled-out  the  initial  questionnaire.  Each  technician  then  performed  the 
baseline  task.  During  the  technicians’  performance,  one  researcher  videotaped  while  the  other  observed 
and  filled-out  the  performance  observation  form.  The  technicians  were  prompted,  as  needed,  to  provide 
think-aloud  protocols  while  performing  their  tasks.  For  the  E&I  Inspection  task,  one  technician 
performed  the  cockpit  inspection  while  the  other  performed  the  engine  compartment  inspection.  Then 
they  switched  locations  on  the  same  aircraft.  For  the  “with  technology”  condition,  technicians  were 
introduced  to  the  prototype  system.  This  consisted  of  a  practice  session  with  both  the  handwriting 
application,  and  the  overall  interface  prior  to  using  the  prototype. 

Upon  completion  of  their  tasks,  the  post-questionnaire  was  administered  to  get  feedback  and  debrief  the 
participants.  On  the  baseline  visit  (without  technology),  the  participants  were  briefed  with  just  the 
purposes  of  the  return  trip  (with  technology)  and  what  activities  they  would  be  expected  to  perform. 
The  debrief  also  set  expectations  with  the  technicians  for  a  follow-up  evaluation  of  improved  prototypes 
in  the  future. 


3.5.2  Shakedown  Event 

In  July  1 998  an  evaluation  event  aimed  at  determining  the  stability  and  maturity  of  the  ITI-ALC  Phase  II 
prototypes  was  held  at  Robins  AFB.  The  USAF  sponsor  who  sent  a  team  to  observe  user  and  prototype 
performance  specified  criteria  for  success  for  this  trial.  In  addition  to  observer  checklists,  video  and 
audiotaping  of  the  training  and  actual  task  performance  trials  of  users  was  done.  Finally,  detailed  time 
coding  of  elapsed  time  to  do  subtasks  was  recorded  for  each  participant. 


3.5.2.1  Shakedown  Procedures 

Each  of  four  participants  completed  a  consent  to  participate  form  and  an  initial  questionnaire  providing 
demographic  information,  data  about  their  prior  experience  in  the  job,  and  self-rating  of  their  previous 
computer  experience.  Then,  using  the  prototype,  each  person  underwent  extensive  practice  to  learn  to 
use  the  handwriting  recognition  software  and  the  pop-up  keyboard  available  at  the  time  of  the 
shakedown.  Once  mastery  of  the  data  input  capabilities  of  the  prototype  were  established,  each 
participant  was  trained  by  an  LMIS  team  member  to  use  the  prototype  for  the  E&I  or  Collaboration 
tasks.  After  completing  the  tutorial  with  assistance,  each  participant  proceeded  to  use  the  prototypes 
independently  and  all  four  participants  did  so.  On  completion  of  the  task,  each  participant  completed  a 
post-questionnaire  indicating  what  they  liked  and  disliked,  and  what  they  would  recommend  changing 
about  the  prototypes. 
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3.5.3  Final  Field  Trial 


Goals 

A  final  field  test  was  conducted  in  Oct  ’98  at  Wamer-Robins  AFB  to  evaluate  PDM  efficiency, 
operating  costs  and  technician  performance  while  using  the  ITI-ALC  system.  Thorough  evaluation  of 
the  ITI-ALC  system  warranted  inclusion  of  several  field  test  goals.  Additionally,  these  goals  had  to  be 
evaluated  under  certain  field  test  constraints.  The  goals  were: 

•  Validate  the  concept  of  an  ITI-ALC  system 

•  Evaluate  the  ITI-ALC  system  in  its  ability  to  address  the  Business  Process  Improvements 
identified  from  Phase  I  activities  (i.e.,  its  ability  to  lower  operating  costs) 

•  Evaluate  the  ability  of  an  ITI-ALC  system  to  improve  job  performance 

•  Identify  specific  improvements  that  can  be  made  to  the  hardware  and  software  for  future 
ITI-ALC  system  implementation  programs. 

•  Provide  lessons  learned  for  development  of  an  ITI-ALC  system. 

Developing  and  conducting  research  in  an  applied  setting,  such  as  a  working  ALC,  often  has  unique 
constraints.  These  constraints  apply  to  field  test  procedures,  data  collection  activities,  time 
considerations,  and  subject  pool.  For  this  field  test,  operational  constraints  included: 

•  Limited  access  to  aircraft  locations  (e.g.,  climbing  on  top  of  the  aircraft). 

•  Time  measurements  were  not  taken  due  to  the  fact  that  demonstration  tasks  were  merely 
segments  of  a  process — total  process  time  could  not  be  measured. 

•  Due  to  time  constraints,  two  simpler  applications  were  developed  for  the  demonstration.  An 
Evaluation  and  Inventory  Inspection  (E&I)  application  and  a  Collaboration  202  Form 
application. 

•  Participants  used  for  the  field  test  were  selected  by  and  asked  to  participate  by  their 
supervisor. 

METHODS 

Participants 

The  ITI-ALC  field  test  included  eleven  participants.  Three  inspectors  used  the  ITI-ALC  system  to 
perform  E&I  inspections.  Six  sheet  metal  mechanics  used  the  ITI-ALC  system  for  the  Collaboration 
application — to  generate  a  202 A  request  form.  Two  engineers,  one  a  supervising  engineer,  used  the  ITI- 
ALC  notebook  system  to  generate  a  response  to  the  202A  form  (i.e.,  the  202B  form). 

Eight  of  the  eleven  participants  participated  in  a  previous  ITI-ALC  prototype  demonstration.  One 
inspector  and  two  mechanics  had  not  used  the  system  prior  to  this  field  test. 

Inspector’s  years-of-experience  ranged  from  VA  to  13  years.  Mechanic’s  had  between  10  and  17  years 
of  experience.  Engineers  declared  between  1 1  and  14  years  of  experience  on  the  job. 

For  the  eleven  participants,  one  mechanic  and  both  engineers  indicated  that  they  had  quite  a  bit  of 
experience  with  computers.  The  remaining  eight  had  no  or  very  limited  experience  with  computers. 
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Apparatus 

The  system  consisted  of  a  server  and  several  mobile  computer  systems,  which  network  to  the  server  via 
wireless  access  points.  The  server  system  supports  servers  for  HTTP,  FTP,  SMTP,  and  POP3 
connections  and  traffic,  and  for  database  connectivity.  The  mobile  systems  connect  to  the  network 
directly  via  the  wireless  access  points.  The  wireless  systems  require  support  for  all  of  these  protocols. 

Hardware 

A  portable  system  has  been  configured  as  a  server  for  off-site  demos.  The  system  is  a  Dell  Inspiron 
M233XT,  and  is  configured  as  follows: 

•  Pentium  23 3Mhz  processor 

•  48MB  RAM 

•  1 1/20X  CD  ROM 

•  3.2  GB  HD 

•  1 3.3”  XGA  display 

•  Sound  system  with  built-in  microphone  and  speakers 

•  Proxim  PC  card  2.4  GHz  radio  w/snap-on  antenna 

Clients.  The  hardware  for  the  ITI-ALC  clients,  Fujitsu  Stylistic  1200  pen  tablets,  were  as  follows: 

•  Pentium  120MHz  processor 

•  32MB  RAM 

•  1.6  GB  HD 

•  8”  Thin  Film  Transistor  (TFT)  color  display 

•  Sound  system  with  built-in  microphone  and  speakers 

•  Proxim  PC  card  2.4  GHz  radio  w/snap-on  antenna 

•  ViCAM  PC  digital  camera 

•  iButton  reader 


Software 


Server.  The  portable  server  operating  system  is  Windows  NT  server,  version  4.0,  with  service 
pack  3.  option  pack  4.  Additional  software  packages  installed  included: 

•  Internet  Information  Server  (IIS)  2.0 

•  Oracle  Server  V7.3 

•  Live  Software  JRunllS 

•  Ipswitch  Imail 

•  Internet  Explorer  4.0,  Service  Packl 

•  ITI-ALC  server  application  software  and  database 

•  Adobe  Exchange  3.0  with  activeX  plugin  and  compose 

•  Proxim  RF  PC  card  drivers 

•  NeoMagic  Display  drivers 

•  CrystalWare  Audio  drivers 

•  RangeLan2  utilities 
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Clients.  The  system  software  for  the  ITI-ALC  clients  is  Windows  95.  Additional  software  installed  on 
the  clients  included: 

•  Pen  services  for  Win95 

•  CIC  handwriting  recognition 

•  Neo-Magic  video  drivers 

•  ESS  sound  drivers 

•  RangeLan  PC  card  drivers 

•  Adobe  Exchange  3.0  with  active  X  plugin 

•  Internet  Explorer  4.0,  Service  Packl 

•  Eudora  light  mail  client 

•  ViCAM  PC  digital  camera  drivers 

•  iButton  server  software 

•  ITI-ALC  client  application  software 

Data  Collection  Materials 

Four  data  collection  techniques  were  used  to  collect  subjective  information  from  test  participants.  The 
first  technique,  a  questionnaire,  targeted  specific  questions  relevant  to  the  ITI-ALC  system.  The 
questionnaire  was  divided  into  three  general  areas:  the  ITI-ALC  concept,  the  perceived  effect  of  an  ITI- 
ALC  system  on  job  performance,  and  usability  of  the  hardware  and  software.  The  second  technique  was 
observation.  As  participants  used  the  system,  test  administrators  collected  notes  on  how  each  person 
used  the  system  and  areas  where  difficulties  were  encountered.  The  third  technique,  interviews,  enabled 
each  participant  to  provide  detail  on  their  responses  to  the  questionnaire  and  clarification  of  problems 
encountered  during  the  tests.  The  final  technique  required  each  participant  to  “walk-through”  the 
application  using  a  paper  prototype.  As  the  technicians  observed  each  prototype  screen,  they  provided 
comments  to  the  interface. 

Procedure 

Upon  arrival,  subjects  signed  an  AFRL/HESR  consent  form.  Next,  they  participated  in  a  tabletop 
training  session  in  which  each  of  the  ITI-ALC  screens  was  shown  and  functions  were  identified. 
Participants  were  taught  how  to  use  the  pen  to  interact  with  the  various  ITI-ALC  screens.  Each 
participant  was  fitted  with  the  ITI-ALC  hardware  to  assure  straps  were  of  proper  length.  Training 
sessions  ranged  from  approximately  1  to  2  hours  per  participant. 

Following  training,  the  inspectors  and  mechanics  proceeded  to  the  hangar  floor  where  they  used  the  ITI- 
ALC  system  to  perfonn  the  E&I  and  Collaboration  tasks  while  at  an  aircraft.  Inspectors  and  mechanics 
were  free  to  pick  areas  on  the  aircraft  to  for  inspection  or  202A  write-ups;  however,  the  ITI-ALC 
scenarios  (e.g.,  only  certain  areas  on  the  aircraft  were  authored  for  the  scenarios),  safety  considerations, 
and  actual  PDM  activities  restricted  aircraft  area  selection.  For  approximately  45  minutes,  the 
inspectors  and  mechanics  used  the  ITI-ALC  system.  Assistance  on  how  to  use  the  ITI-ALC  system  was 
limited  to  two  types.  First,  the  test  administrator  assured  that  participants  used  all  functions  available  in 
the  application.  Second,  if  the  participant  had  trouble  with  a  ftmction,  they  were  given  help  only  as 
required.  Help  was  limited,  in  most  cases,  to  verbal  assistance. 
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Engineers  participated  in  tabletop  use  of  the  ITI-ALC  system  to  perform  the  Collaboration  task.  They 
responded  to  202A  forms  generated  by  the  mechanic  participants  earlier  in  the  field  test.  In  this  way, 
data  on  the  202 A  forms  were  somewhat  representative  of  the  types  of  data  actually  available  through  an 
ITI-ALC  system. 

Following  the  field  test  activities,  participants  filled  out  a  questionnaire  (see  Appendix  C),  discussed 
their  responses  to  the  questionnaire  with  the  test  administrator,  and  commented  on  each  of  the  ITI-ALC 
screens  (Appendix  E  and  Appendix  F). 
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4.  Results  and  Discussion 


4.1  FBE  #  1  Formal  Usability  Study  Event 

FBE  #1  began  the  user-centered,  iterative  development  cycle  for  ITI-ALC  Phase  II.  During  this  event, 
ten  users  who  varied  in  terms  of  domain  expertise  and  personal  traits  (education,  vision,  physical  size, 
age,  etc.)  provided  updates  to  the  ITI-ALC  Phase  I  user  requirements  and  commented  on  potential 
technologies  shared  with  them  during  FBE  #1.  (For  a  detailed  account  of  this  activity,  please  see  ITI- 
ALC  Phase  II  FBE  #1  Research  Report,  July  1997  and  the  accompanying  highlights  videotape.)  The  six 
priority  areas  identified  by  FBE  #1  participants  as  those  for  ITI-ALC  Phase  II  prototyping  to  address 
were: 

•  Sign-on  and  sign-off  using  smart  cards  with  secure  PIN  numbers; 

•  IETMs  and  TO’s  with  current  data  for  specific  aircraft; 

•  Communications/collaboration  for  interaction  with  engineers  or  remote  experts; 

•  Electronic  parts  tracking,  especially  for  back-shop  coordination/status; 

•  Electronic  support  for  unpredictables,  for  entering  information  and  getting  it  quickly  into  PDM,  and 

•  Inspection  support  for  easy  defect  entry  and  communication  for  PDM  action. 

This  initial  formal  usability  event,  and  its  findings,  served  as  the  key  inputs  for  the  initial  prototyping 
effort. 
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4.2  Informal  Data  Collection  Events 


As  noted  earlier,  two  data  collection  efforts  to  WR-ALC  and  one  to  Tinker  AFB  followed  to  conduct 
additional  research  about  the  nature  and  relevance  of  these  priority  areas  for  the  F-15  and  other  product 
lines  within  the  USAF.  First  prototyping  included  examination  of  various  hardware  platforms,  display 
capabilities  and  interface  designs.  In  early  November  1997,  a  second  usability  event  occurred  when  the 
ITI-ALC  Phase  II  team  conducted  an  informal,  small  scale  feedback  session  with  E&I  inspectors,  skin 
mechanics,  engineers,  and  avionics  technicians  at  Robins  AFB  to  elicit  their  reactions  to  early 
prototypes.  (For  a  detailed  account  of  this  event,  please  see  ITI-ALC  Trip  Report  -  Warner  Robins  AFB, 
November  5  &  6,  1997.) 

The  initial  E&I  Inspection  prototype  was  implemented  on  a  Newton  2000  and  we  checked  handwriting 
recognition  with  both  “printing”  (block  letters)  and  “cursive”  options.  Four  E&I  inspectors  were 
individually  walked  through  the  initial  prototype  application  and  gave  us  feedback  about  legibility, 
response  time,  whether  there  was  missing  or  unnecessary  information  on  each  screen,  the 
weight/feel/size  of  the  platform,  etc.  Major  items  we  learned  that  were  considered  for  the  E&I  prototype 
refinement  included: 

(1)  the  E&I  process  was  undergoing  change: 

-  we  needed  to  allow  for  individual  inspector  preferences  and  possible  use  of  rational  criteria 
for  the  order  and  sequence  of  inspections, 

-  the  stand  alone  system  and  clerk  present  at  FBE  #1  were  gone, 

a  direct  connection  to  PDMSS  was  almost  operational  (i.e.,  triple  entry  of  inspection 
information  was  becoming  double  entry  and  the  ITI-ALC  prototype  could  make  it  a  single 
entry  process), 

(2)  the  defect  list  changed: 

-  with  use  by  two  new  inspectors  the  list  was  reduced  to  190,  instead  of  212  defects, 

-  an  initial  cut  was  made  to  specify  the  48  worst  defects  (those  that  take  longest  to  fix,  need 
parts  that  are  hard  to  get,  have  major  impact  on  aircraft  output  dates)  and  was  made  available 
to  the  ITI-ALC  team, 

(3)  the  E&I  and  first  line  supervisors  were  working  toward  more  complete  173’s  (with  more  fields  pre¬ 
filled)  for  mechanics 

-  an  initial  example  was  provided  for  possible  inclusion  in  ITI-ALC  prototyping. 

Prototypes  to  support  the  Collaboration  and  Operational  Check  scenarios  derived  from  the  initial  user 
requirements  both  were  implemented  using  a  Fujitsu  Point  5 1 0  portable  platform  with  handwriting  as 
the  input  medium.  Again  we  sought  user  response  related  to  the  platform  (weight,  feel,  how  they  would 
hold/use  it),  and  asked  about  the  same  characteristics  mentioned  above.  Two  mechanics  and  two 
engineers  had  a  very  positive  reaction  to  the  Collaboration  prototype.  Specific  feedback  from  the 
engineers  included:  (a)  that  the  ITI-ALC  prototype  needed  to  work  with  their  existing  email  systems 
(Microsoft  Exchange)  and  (b)  that  they  needed  much  better  prioritization  information  (indicators  of 
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which  message  to  attend  to  first).  The  mechanics  feedback  included:  (a)  a  concern  that  we  get  planners 
involved  in  feedback  and  their  sign-off/role  to  be  clearer  in  the  application,  (b)  consider  adding  phone 
capability  (for  mechanics  to  call  or  be  called  by  engineers),  and  (c)  the  light  glare  on  the  screen 
interfered  seriously  with  legibility  and  use  of  this  initial  prototype.  All  of  this  feedback  was  considered 
in  the  changes  and  refinements  made  to  prototypes  during  December  1997  through  February  1998.  Due 
to  difficulties  in  obtaining  needed  technical  documentation  and  other  resource  constraints  associated 
with  developing  the  Operational  Check  scenario,  the  ITI-ALC  Phase  II  team  prototyping  effort  was 
focused  on  the  E&I  Inspection  and  Engineering  Assistance  (Collaboration)  prototypes  as  of  January, 
1998. 


4.3  FBE  #2  Formal  Usability  Study  Event 

For  FBE  #2,  conducted  in  February  and  March  1998,  two  of  the  Robins  AFB  Evaluation  and  Inventory 
Inspectors  participated  in  the  study  (two  other  inspectors  were  unable  to  participate  due  to  extended 
absence  from  work  for  health  problems).  First,  the  overall  results  of  the  FBE  #2  E&I  inspection  effort 
are  summarized.  Then,  a  detailed  descriptive  analysis  of  the  errors,  information  needs,  and  physical 
interactions  of  the  inspectors  while  trying  out  the  prototype  system  is  provided.  Finally,  the  inspectors’ 
preferences  related  to  the  prototype  system  are  presented. 

Similarly,  we  report  the  results  of  the  Collaboration  prototype  evaluation  that  was  tried  out  by  six  F-15 
skin  mechanics  and  one  structural  engineer. 


4.3.1  FBE  #2  Demographic  Summary 

The  two  E&I  inspectors’  demographic  characteristics  at  the  time  of  FBE  #2  included:  Participant  1 1 
(PI  1)  who  had  more  than  32  years  of  aircraft  maintenance  experience  and  8.5  years  experience  doing 
E&I  inspections  and  Participant  14  (PI  4)  who  had  13  years  of  aircraft  maintenance  experience  and  just  9 
months  of  experience  doing  E&I  inspections.  Their  respective  ages  were  57  and  40  years.  Both  men 
had  no  prior  experience  with  any  kind  of  computer.  Both  men  used  corrective  lens  (PI  1  had  bifocals 
and  P14  was  myopic)  and  each  reported  a  problem  with  sensitivity  to  bright  light. 

Both  participants  did  the  E&I  inspection  of  the  cockpit  (areas  2,  2A)  and  the  engine  compartments  (area 
12)  without  the  prototype  in  February  1998  and  with  it  in  March  1998  and  they  identified  several  areas 
for  improvement  during  the  field  usability  evaluation. 

The  six  mechanics  in  FBE  #2  were  Participants  4,  15,  17,18,  19,  and  20.  They  had  an  average  of  10.7 
years  experience  doing  aircraft  skin  repair  work.  Their  average  age  was  40  years.  Four  of  these 
participants  had  no  prior  experience  using  a  computer,  while  two  of  them  had  moderate  experience, 
having  used  a  home  PC  to  access  the  Internet,  prepare  documents  with  Microsoft  Word,  and  for  email. 
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Three  of  the  skin  mechanics  had  no  eye  correction  (reported  that  they  have  20/20  vision).  Participant  4 
wore  corrective  lenses  for  myopia  and  Participants  17  and  20  had  bifocal  correction. 

The  structural  engineer,  Participant  16,  had  more  than  25  years  of  experience  working  with  F~15s  and  he 
was  56  years  old  when  FBE  #2  was  conducted.  He  reported  that  his  experience  using  computers  was 
“moderate"  having  done  some  programming  and  used  a  PC  with  Win95  and  for  email.  He  used  eye 
correction  for  myopia  and  bifocals. 

Each  of  the  six  individuals  participated  in  the  “without”  and  “with  technology”  trials  of  the  prototypes  in 
February  and  March  of  1998.  They  also  identified  several  areas  for  improvement,  which  are 
summarized  in  this  report.  Their  inputs  were  used  to  modify  the  prototype  for  the  shakedown  and  final 
evaluation  efforts. 


4.3.2  FBE  #2  Errors,  Failures,  Requests  for  Information,  and  Physical  Interactions 

These  category  structures  were  created  from  the  measurement  needs  (e.g.  observing  mispressed  buttons 
for  usability  concerns)  outlined  in  the  FBE  #2  demonstration  plan  (ITI-ALC  Technical  Report; 
Document  #63A1 56465,  Rev:  New,  4  March  1998).  Additional  categories  were  added  to  cover  other 
observed  relevant  behaviors  (e.g.  from  watching  the  tapes  it  became  clear  that  one  important  request  for 
information  was  verification  (e.g.  "Am  I  going  the  right  way?")  so  an  explicit  subcategory  was  created 
within  “requests  for  information”  for  this.) 

Tables  are  presented  for  each  of  these  error  categories  to  summarize  the  frequency  of  each  phenomenon. 
The  E&I  task  tables  are  organized  by  E&I  participant  (PI  1  or  P14  for  the)  and  task  (cockpit  or  engine 
compartment  inspection).  Note  that  due  to  counterbalancing  for  this  study,  PI  1  did  the  cockpit  first  and 
the  engine  compartments  second,  while  P14  did  the  engine  compartment  first  and  the  cockpit  second. 
Hence,  any  phenomena  due  to  lack  of  practice  will  attenuate  oppositely  across  the  tasks  for  the 
participants.  The  participants  organize  the  Collaboration  task  tables. 


4.3.2.1  FBE  #2  Categorization  of  Errors  and  Failures 

Errors  and  failures  were  coded  using  the  following  categorization  scheme: 

•  Any  repeated  button  presses  or  check  box  taps  (due  to  system  not  registering  tap  or  linking  problem) 

•  Handwriting  recognition  errors  (e.g.  Ben5  instead  of  Ben,  etc.)  Scrolling  problems 

•  Hitting  wrong  key  (e.g.  hitting  "edit"  key  instead  of  backspace  on  editing  palette) 

•  Interface  object  interference  (e.g.  edit  palette  on  top  of  exit  button) 

•  Misinterpretation  of  what  something  means 

•  Slow  to  perceive  or  forgets  needed  information 
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•  Finds  information  not  pertaining  to  correct  region  of  aircraft 

•  Wireless  network  failure 

•  Interface  performance  failure 

The  most  common  problems  in  this  category  for  E&I  inspection  (Table  4.3.2. 1-1)  were  repeating  actions 
(11),  noting  electronic  173  cards  in  wrong  region  (11),  tapping  while  scrolling  (instead  of  just  holding 
the  arrow  down)  (7),  and  having  problems  with  handwriting  (6).  The  repeated  action  problem  and  the 
scrolling  problem  were  very  likely  due  to  two  sub-problems.  First,  the  screen  of  the  prototype  client  is 
highly  sensitive  to  motion  -  it  makes  a  distinction  between  a  “tap”  and  a  “drag.”  If  a  participant  moved 
the  pen  on  the  screen  while  tapping  a  checkbox  (e.g.  by  using  a  metaphor  from  paper  and  trying  to 
actually  check  it  or  fill  it  in),  the  system  would  not  register  the  tap.  Secondly,  the  occasional  slowness  in 
response  caused  participants  to  sometimes  believe  that  their  valid  taps  had  not  registered.  For  the 
scrolling  case,  it  is  likely  that  a  similar  phenomenon  arose  -  in  this  case,  because  the  system  was  not 
immediately  responsive,  Pll  may  have  developed  a  tapping  strategy.  In  addition,  he  repeatedly  tried  to 
drag  the  arrow  button,  indicating  that  he  did  not  quite  understand  how  the  scrolling  panel  operated. 
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Table  4.3.2.1-1  FBE  #2  E&  I  Inspection  Task  Frequency  of  Errors  and  Failures 


Pll 

P14 

Total 

Errors  and 

Failures 

Cock¬ 

pit 

Engine 

m 

RSI 

a)  Repeated  Actions 

5 

1 

3 

2 

8 

3 

11 

b)  Handwriting 

5 

0 

1 

5 

1 

6 

c)  Scrolling 

o 

J 

2 

2 

3 

4 

7 

d)  Wrong  Key 

o 

J 

0 

0 

3 

3 

e)  Interference 

0 

1 

0 

1 

1 

f)  Misinterpretation 

2 

0 

2 

0 

2 

g)  Slow  Perception 

1 

2 

2 

1 

4 

5 

h)  Region  Problem 

2 

4 

4 

1 

6 

5 

11 

i)  Network  Failure 

0 

0 

2 

2 

2 

2 

4 

j)  Interface  Failure 

0 

0 

1 

0 

1 

1 

Total 

19 

9 

12 

11 

31 

20 
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The  most  common  problems  experienced  by  the  users  of  the  Collaboration  (Table  4.3.2. 1-2), 
(Engineering  Assistance)  prototype  were:  difficulty  with  handwriting  (14),  difficulty  remembering 
needed  information  (11),  and  hitting  the  wrong  key  or  area  of  the  interface  (9).  Individuals  had 
difficulty  with  particular  letters,  e.g.,  “f  ’  which  lead  them  to  need  to  erase  and  retry  information  entry. 
The  Form  202A  completion  activities  were  affected  by  the  same  kind  of  handwriting  problems  noted 
above  for  the  E&I  inspection  prototype. 


Table  4.3.2.1-2  FBE  #  2  Collaboration  Task  Frequency  of  Errors  and  Failures 


Errors  and  Failures 

P15  P17 

P18 

P19 

P20 

P4 

Total 

a)  Repeated  Actions 

3 

3 

b)  Handwriting 

2 

6 

2 

4 

14 

c)  Scrolling 

1 

1 

d)  Wrong  Key 

1 

3 

2 

1 

2 

9 

e)  Interference 

0 

f)  Misinterpretation 

0 

g)  Slow  Perception 

2 

4 

2 

2 

0 

10 

h)  Region  Problem 

0 

i)  Network  Failure 

1 

1 

2 

j)  Interface  Failure 

2 

2 

0 

4 

Total 

4 

12 

9 

4 

4 

10 

43 
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4.3.2. 2  FBE  #2  Categorization  of  Requests  for  Information 

The  requests  for  information  were  coded  using  the  following  categorization  scheme: 

•  Interface  widget  location  information  (e.g.,  Where  is  button  X?) 

•  "How  to"  information  (e.g..  How  to  I  get  back  to  other  region) 

•  General  confusion  (e.g.,  What  do  I  do  next?  I  don't  know  what  to  do  here) 

•  Verification  of  information  (e.g.,  "Is  that  right?"  “Should  I  log  on  here?”) 

•  Explanation  of  behavior  of  technology  (e.g.,  “Why  did  it  do  that?") 

The  most  common  requests  were  for  the  E&I  prototype  (Table  4.3.2.2-1)  were  for:  verification  (15), 
“How  to”  information  (12)  and  for  explanation  (11).  The  “how  to”  requests  are  for  specific  information 
on  how  to  accomplish  something  (e.g.  “how  do  I  backspace?”  “What  do  I  do  to  get  to  the  other 
region?”).  Some  of  these  actions  were  covered  in  training  -  the  requests  point  to  places  where  the 
interface  does  not  make  readily  obvious  how  to  accomplish  the  action  (i.e.  the  interface  is  not  “walk  up 
and  use”  for  that  action).  Verification  of  information  serves  a  similar  purpose  -  in  these  cases  the 
participants  have  an  idea  of  what  to  do  (which  may  be  right  or  wrong)  but  are  not  confidant  enough  to 
take  the  action  without  further  reassurance.  Asking  for  explanation  is  a  case  where  the  system  is  not 
making  its  behavior  clear  enough,  e.g.  P14  asks  about  the  hourglass.  This  is  a  standard  icon  in  Windows 
indicating  the  system  is  working  on  something  so  the  user  should  wait.  Even  though  P 1 4  was  exposed 
to  this  information  during  training,  the  hourglass  image  was  not  enough  of  a  reminder. 


Table  4.3.2.2-1  FBE  #2  E&I  Task  Frequency  of  Requests  for  Information 


Pll 

P14 

Total 

Requests  for 
Information 

a)  Location 

2 

0 

0 

1 

2 

1 

3 

b)  “How  To” 

7 

0 

2 

3 

9 

3 

12 

c)  General 

3  1 

3 

2 

3 

5 

6 

11 

d)  Verification 

3 

6 

5 

1 

8 

7 

15 

e)  Explanation 

2 

1 

0 

5 

2 

6 

8 

Total 

17 

10 

9 

13 

26 

23 
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The  most  common  requests  for  the  Collaboration  prototype  (Table  4.3.2.2-2)  users  were:  for 
verification  (27),  “how  to”  information  (16),  and  general  confusion  (14),  e.g.,  being  unsure  about 
leaving  fields  in  the  form  blank  or  about  how  to  use  the  camera.  There  were  a  few  instances  of  asking 
for  an  explanation  (5),  e.g.,  asking  whether  the  aircraft  will  block/interfere  with  the  wireless  connection 
or  asking  whether  a  field  is  big  enough  to  handle  an  entry.  Users  of  this  prototype  made  no  requests  for 
locational  information. 


50 


Table  4.3.2.2-2  FBE  #  2  Collaboration  Task  Frequency  of  Requests  for  Information 


Requests  for  Information 

a)  Location 

0 

b)  “How  To” 

3 

4 

4 

1 

2 

2 

16 

c)  General 

1 

1 

5 

1 

4 

2 

14 

d)  Verification 

3 

8 

7 

4 

o 

2 

27 

e)  Explanation 

2 

2 

1 

5 

Total 

7 

15 

18 

6 

10 

6 

62 

4.3.2.3  FBE  #2  Categorization  of  Physical  interactions 

The  physical  interactions  were  coded  using  the  following  categorization  scheme: 

•  Any  adjustments  or  accommodations  made  while  wearing  the  Fujitsu  1200 

•  Moving  1200  out  of  way  (but  keeping  on  body) 

•  Taking  unit  off 

•  Comments  relating  to  this:  e.g.  "This  feels  kind  of  awkward" 

•  Stowing  pen  when  1200  not  in  active  use 

•  Hand  cleaning  required  before  using  device 

•  Effects  of  being  left/right-handed  on  use  of  system 

The  most  common  E&I  prototype  (Table  4.3.2.3-1)  physical  interaction  issues  were:  the  need  to  adjust 
the  system  (5),  removing  the  unit  (3),  and  commenting  on  the  system  getting  in  the  way  (2).  PI  1  clearly 
had  a  problem  with  the  form  factor  and  support  structure  for  the  system.  During  the  cockpit  inspection 
he  took  the  unit  off  for  substantial  periods  of  time,  and  commented  once  when  he  had  it  on  that  it  was 
awkward.  This  was  during  an  inspection  under  the  dashboard  in  the  forward  cockpit  where  he  had  to 
hold  the  Fujitsu  1200  out  of  the  way  with  one  hand  while  holding  a  flashlight  with  the  other.  His 
strategy  of  interleaving  the  inspection  task  with  using  the  system  appeared  to  make  it  especially  hard  for 
him.  After  doing  a  short  piece  of  the  inspection  he  had  to  pick  up  the  device  and  use  it,  and  then  let  it 
hang  down,  move  it  to  the  side  or  take  it  off  for  the  next  part  of  the  inspection.  He  had  similar  problem 
in  the  engine  compartment  inspection.  P14  seemed  to  have  less  of  a  problem,  but  still  had  a  problem. 
No  explicit  interaction  problems  were  observed  where  he  had  to  take  off  the  unit  or  move  it  out  of  the 
way  while  in  the  cockpit,  though  he  did  comment  that  he  thought  it  was  going  to  get  in  the  way.  There 
were  two  times  during  the  engine  compartment  inspection  that  he  had  to  adjust  the  system.  His  easier 
time  may  be  due  to  his  strategy  of  doing  the  entire  inspection  first  while  having  the  unit  hanging  down, 
and  then  using  the  system  in  one  block  when  he  finished  the  inspection.  Also,  because  he  was  a  smaller 
individual  he  might  be  able  to  accommodate  the  form  factor  more  easily.  The  preference  data  described 
in  the  next  section  address  participants’  concerns  on  the  form  factor  as  well. 
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Table  4.3.2.3-1  FBE  #2  E&  I  Task  Frequency  of  Physical  Interaction  Problems 


Pll 

P14 

Total 

Physical 

a)  Adjustment 

1 

2 

0 

2 

1 

4 

5 

b)  Removing  Unit 

3 

0 

0 

0 

3 

0 

3 

c)  Comments 

1 

0 

1 

0 

2 

0 

2 

d)  Stowing  Pen 

0 

1 

0 

0 

0 

1 

1 

e)  Hand  Cleaning  ; 

0 

1 

0 

0 

0 

1 

1 

f)  Left/Right  Hand 

0 

1 

0 

0 

0 

1 

1 

Total 

5 

5 

1 

2 

6 

7 

13 

Overall  there  were  not  many  issues  (9)  raised  about  physical  interaction  issues  for  the  Collaboration 
prototype  (Table  43.2.3-2).  The  most  common  physical  interaction  issues  were  related  to  taking 
pictures  (4)  where  users  dropped  the  pen  while  using  the  camera  or  had  shaking  hands  while  trying  to 
control  the  camera  in  mid-air.  Three  instances  occurred  where  users  had  difficulty  with  screen  glare. 
Additionally,  one  participant  had  difficulty  stowing  the  pen,  and  another  participant  had  trouble  with  the 
strap  getting  in  his  way  while  trying  to  move  the  pen.  The  results  of  the  behavioral  coding  described 
here  are  summarized  in  the  next  section. 


Table  4.3.2.3-2  FBE  #  2  Collaboration  Task  Frequency  of  Physical  Interaction  Problems 


Physical 

a)  Adjustment 

b)  Removing  Unit 

c)  Comments 

d)  Stowing  Pen 

e)  Hand  Cleaning 

f)  Left/Right  Hand 

g)  Taking  picture 
Total 


1 

2  1 
1 


1  1  1 

0  14  3 


1 

1  0 


1 

0 

3 
1 
0 
0 

4 
9 
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4.3.3  FBE  #  2  Preference  Data 


Both  the  pros  and  cons  of  participants’  experience  in  evaluating  the  prototypes  are  provided  here.  The 
inspector's  post-questionnaires  are  presented  first.  These  results  suggest  that  they  are  positive  about  the 
use  of  the  prototype  system.  This  is  followed  by  a  summary  of  the  post-trial  interview  and  debriefing 
feedback  received  by  the  ITI-ALC  team.  Then,  material  from  the  questionnaires  and  interviews  is 
summarized  for  the  skin  mechanics  regarding  their  feedback  on  the  Collaboration  system. 


4.3.3. 1  FBE  #  2  E&l  Inspection  Prototype  Feedback 

When  asked  what  they  liked  best,  Pll  noted,  “It  was  fairly  easy  to  use,  but  somewhat  difficult  when 
making  a  new  write-up.  PI 4  stated  that  mobile  system  “saves  time,  gets  information  to  other 
organizations  much  quicker,  and  saves  on  paperwork.”  Both  men  noted  that  the  defects  from  their 
master  listing  were  in  the  system  and  they  did  not  specify  any  additional  information  they  need  to  do 
their  tasks  with  this  prototype. 

When  asked  what  they  liked  least  about  using  the  mobile  prototype,  PI  1  said  “somewhat  difficult  when 
making  a  new  write-up”  while  PI 4  said  he  “worries  about  dropping  or  bumping  the  unit  during 
inspections.  Changes  the  inspectors  would  like  to  see  in  the  near  future  include:  Pll:  bigger  screen, 
fewer  handwriting  recognition  errors,  and  more  training  time,  and  PI 4:  faster  response  time,  ability  to 
partially  change  defect  descriptions,  and  more  protection  around  the  unit.  Comparing  use  of  the 
prototype  to  doing  this  task  using  paper,  the  inspectors  had  very  different  reactions.  Pll  stated  that  “it 
(the  prototype)  slowed  me  up  approximately  25%  of  normal  time  by  not  being  used  to  it.”  P14  said, 
“Great;  I  think  this  will  be  a  great  asset  to  the  Air  Force.” 

PI  1  indicated  that  he  could  see  the  text  and  graphics  in  the  application  and  find  the  information  needed 
for  the  E&I  inspection  very  easily.  P14  also  rated  seeing  text  as  very  easy,  but  he  did  not  express  an 
opinion  about  graphics  and  he  ranked  finding  the  information  he  needs  for  an  E&I  inspection  as  a  “3”  on 
a  5-point  scale  where  “1”  means  “very  easy”  and  “5”  means  “very  difficult.” 

When  asked  on  the  questionnaires  about  anything  else  to  share  with  the  ITI-ALC  team  about  their 
experience  with  the  mobile  prototype,  Pll  said  “I  would  like  more  meeting  sessions  with  evaluators  and 
computer.  P14’s  response  was  “I  know  nothing  about  computers,  but  this  unit  was  easy  to  leam  and  it 
seems  to  me  it  can  do  many  things,  I  just  have  to  leam  how  to  operate  it.” 

During  interviews,  inspectors  rated  the  physical  prototype  (Fujitsu  1200)  and  the  application  running  on 
it  as  follows: 

Attributes:  PI  1  P14 

Computer  weight:  “O.K.”  “Lighter  than  expected” 

Seeing  infomiation  on  screen  “very  clear”  “very  clear” 
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Using  the  pen  to  write  on  the  computer  “difficult”  “difficult” 

Correcting  handwriting  mistakes  “difficult”  “O.K.” 

Additional  remarks,  which  add  to  the  information  that  the  inspectors  shared  on  their  questionnaires,  are 
summarized  here.  PI  1  noted  that  right  now  paper  would  be  better  for  writing  new  cards,  and  said  if  they 
hadn’t  used  the  computer  for  writing  new  cards  then  it  (the  computer)  would  probably  be  better  than 
paper.  P14  commented  that  he  hoped  technicians  would  be  offered  the  opportunity  to  receive  some 
schooling  to  learn  how  to  use  mobile  tools.  Both  inspectors  noted  that  there  was  some  defect  to  area 
mappings  that  didn’t  pertain  to  the  areas  they  inspected.  Their  input  on  this  matter  was  recorded 
immediately  after  the  debriefing  sessions. 


4.3.3.2  FBE  #  2  Collaboration  (Engineering  Assistance)  Prototype  Feedback 

Participants  indicated  that  what  they  liked  most  was  having  the  information  “at  your  fingertips”,  getting 
information  and  answers  faster,  and  getting  away  from  using  paper.  What  they  liked  least  was 
handwriting  on  the  screen,  size  of  the  mobile  computer,  and  presence  of  too  many  cords.  Participant  15 
indicated  a  need  for  TO’s  that  were  not  available  to  him  while  doing  the  vertical  tail  assembly  task. 
(Please  note  that  there  were  five  responses  reported  in  this  section  of  the  report  because  Participant  4  did 
not  complete  the  post-task  activities  due  to  time  constraints  in  the  data  collection  effort  for  FBE  #2.) 

Participant  ratings  indicated  that  they  found  it  very  easy  to  find  information  needed  for  Form  202A 
completion.  All  participants  except  Participant  1 8  also  found  it  very  easy  to  see  textual  material  while 
using  the  prototype.  Three  participants  indicated  that  seeing  graphic  information  while  using  the 
prototype  was  very  easy,  while  two  participants  indicated  that  it  was  difficult.  (Note  that  one  of  these 
participants  did  not  have  corrective  lenses  while  the  other  participant  is  nearsighted  and  wears  bifocals.) 

When  asked  what  they  would  change  to  improve  the  prototypes,  three  skin  mechanics  stated  that  they 
would  improve  the  handwriting  recognition/reduce  the  number  of  errors  for  future  prototypes  and  one 
participant  indicated  that  having  larger  letters  on  the  pop-up  keyboard  would  be  helpful.  One  participant 
also  mentioned  that  having  a  smaller  unit  would  be  preferred.  Two  participants  indicated  that  they 
would  not  change  any  features  of  the  prototype  as  of  FBE  #2. 

All  five  participants  made  positive  statements  about  their  feelings  using  the  mobile  computer  instead  of 
paper  to  do  their  tasks.  Three  of  these  five  mechanics  did  state  that  they  felt  slightly  awkward  at  first  but 
that  they  felt  that  with  practice  they  could  become  quite  comfortable  using  such  a  prototype.  One  of  the 
mechanics  stated,  “It  was  excellent.  Mechanic  has  too  much  paper  work  already”  (PI 5). 
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4.4  Shakedown  Event 


The  Shakedown  event  followed  in  the  cycle  of  iterative  development.  Many  of  the  system  and  interface 
challenges  that  surfaced  during  FBE  #2  were  addressed  successfully  by  the  ITI-ALC  Phase  II  team.  The 
team  also  prepared  and  delivered  very  successful  training  that  resulted  in  the  incidence  of  errors  and 
failures,  requests  for  information,  and  physical  problems  to  be  dramatically  reduced. 


4.4.1  Shakedown  Preference  Data 

The  four  participants  completed  the  same  post-trial  questionnaire  as  the  one  used  in  FBE  #2.  Overall 
their  responses  were  very  positive  regarding  their  use  of  the  prototype.  These  completed  questionnaires 
are  included  in  Appendix  D. 

Their  rankings  of  ease  of  seeing  and  finding  information  were  all  “1”  or  “2”  on  the  scale  of  1  to  5  with 
“1”  being  “very  easy”  and  “5”  being  very  difficult,  with  one  exception.  Participant  2  ranked  seeing  the 
graphic  representation  of  the  aircraft  as  a  “5”  for  the  Collaboration  task. 

All  participants  made  suggestions  for  changes  and  improvements  for  the  prototypes  that  mentioned  the 
pen  and  its  sensitivity  and  they  suggested  developers  should  try  to  reduce  its  sensitivity.  Participants  2 
and  4  both  mentioned  faster  response  times  as  desirable.  Participant  1  suggested  that  a  smaller  unit 
might  be  helpful  for  inspection  tasks  or  tasks  in  tight  spots.  Participant  2  commented  on  the  tediousness 
associated  with  continuous  use  of  the  pen.  Participant  3  suggested  that  a  larger  screen  would  help  or 
that  it  would  be  useful  to  enable  technicians  to  adjust  the  size  of  the  field(s)  they  are  currently  using. 
Also  Participant  3  noted  that  the  mechanic  should  get  some  concrete  feedback  about  the  voice  and 
picture  attachments  before  sending  the  Form  202  A  to  the  engineer.  Participant  3  also  commented  about 
the  bungee  cord  for  the  pen  wrapping  around  the  device.  Participant  4  noted  that  the  prototype  should 
“find  a  way  to  let  technician  know  if  they  were  about  to  exit  without  submitting  defects  recorded  during 
that  session”. 

Comparing  use  of  the  prototype  to  using  paper  for  these  tasks,  all  participants  strongly  preferred  the 
prototype.  Though  one  (P2)  expressed  concern  about  having  a  very  reliable  system.  Their  overall 
comments  about  the  experience  were  very  positive.  PI  stated,  “I  enjoyed  taking  part  in  this  testing  of 
the  mobile  system.  I  found  it  very  simple  and  very  useful  and  hope  to  use  this  system  in  the  future.”  P2 
commented,  “It  was  fun,  I  enjoyed  it,  a  good  system.”  P3  stated,  “Much  easier  and  handier  than  utilizing 
the  3  or  4  items  that  this  unit  replaces.”  Finally,  P4  commented,  “Loved  it.  I  found  it  very  useful  as  a 
guide  of  how  to  perform  E&I  inspections.  For  non-experienced  inspectors,  this  would  be  a  great  tool 
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4.5  Final  Field  Trial 


RESULTS 

Materials  used  to  collect  data  included  the  questionnaire,  interviews,  observations  (Appendix  E)  and 
screen  walkthroughs  (Appendix  F).  Appendix  E  presents  each  questionnaire  item,  the  responses  (in  a 
graph),  comments  by  the  participants  (in  italic  bold),  and  test  administrator  observations  [identified  in 
brackets].  Appendix  F  represents  the  screen  walkthrough  participants.  These  walkthroughs  are  divided 
into  three  applications:  inspectors,  mechanics,  and  engineers.  Again,  participant  comments  are  in  italic 
bold  and  test  administrator  observations  are  [identified  in  brackets]. 


DISCUSSION 

The  purpose  of  this  field  test  was  to  achieve  the  goals  listed  below.  In  order  to  achieve  these  goals 
within  the  constraints  of  the  project,  data  collection  was  completed  using  four  techniques.  This 
Discussion  section  is  organized  in  accordance  with  the  project  goals  and  uses  data  presented  in  the 
Results  section  to  address  each  of  those  goals. 

Project  Goals 

Goal  1.  Validate  the  concept  of  an  ITI-ALC  system 

Goal  2.  Evaluate  the  ITI-ALC  system  in  its  ability  to  address  the  Business  Process 

Improvements  identified  from  Phase  I  activities  (i.e.,  its  ability  to  lower  operating  costs) 
Goal  3.  Evaluate  the  ability  of  an  ITI-ALC  system  to  improve  job  performance 
Goal  4.  Identify  specific  improvements  that  can  be  made  to  the  hardware  and  software  for  future 
ITI-ALC  system  implementation  programs. 

Goal  5.  Provide  lessons  learned  for  development  of  an  ITI-ALC  system. 

Goal  1 .  ITI-ALC  Concept  Validation 

Participants  validated  the  ITI-ALC  concept  in  several  ways.  They  indicated  that  this  system  would 
increase  access  to  information  and  that  with  practice  and  computer  experience,  novices  could  use  it 
easily.  In  addition,  technicians  indicated  that,  depending  on  the  task  at  hand;  they  would  use  the  system 
for  their  primary  job.  They  also  indicated  that  they  would  use  an  ITI-ALC  system  fairly  frequently  if 
provided. 

Participants  stated  that  this  system  would  be  useful  in  numerous  maintenance  areas,  including  “all  depot 
maintenance  on  aircraft”  and  “part  numbers,  repairs,  general  information-TOs.”  Maintenance  areas  where  they 
did  not  think  it  would  be  useful  included  “tight  areas.” 

Goal  2.  Ability  to  Address  the  Business  Process  Improvements 

Five  Business  Process  Improvements  (BPIs)  were  addressed  through  data  collection  activities  during  the 
ITI-ALC  field  test.  For  specific  information  concerning  the  full  implication  of  each  BPI  refer  to  the 
Architecture  Report  (Systems  Research  and  Application  Corporation,  1995). 
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BPI 8.  Electronic  Signatures 

Participants  felt  the  I-button  (electronic  signature  emulator)  was  faster  and  more  secure  than  the  current 
signature  process  (Job  Performance  questions  8  &  9).  It  was  observed  that  the  I-button  was  easy  to 
disconnect  and  therefore  frequently  fell  off.  To  implement  the  I-button,  modifications  would  need  to  be 
made  to  minimize  loss  of  the  I-button  due  to  inadvertent  disconnections. 

BPI  9.  Performance  Metrics  Based  on  Actual  Data 

Reactions  varied  concerning  the  accuracy  of  estimates  of  time  to  complete.  Overall,  participants  did  not 
indicate  that  they  thought  it  would  increase  accuracy  over  other  methods. 

BPI  10.  User  Technical  Information  Presentation  System 

Participants  indicated  that  they  thought  the  ITI-ALC  system  would  increase  use  of  TO  data,  yet  not  for 
all  tasks  (ITI-ALC  Concept  question  1) 

BPI  13.  Automated  and  Integrated  Technical  and  Diagnostics  Information 

In  comparison  with  the  current  process,  participants  felt  that  ordering  parts  would  be  faster  and  easier 
with  an  ITI-ALC  system  (Job  Performance  question  4).  Overall,  they  felt  an  ITI-ALC  system  would 
increase  access  to  technical  information  (ITI-ALC  Concept  question  2). 

BPI  14.  Multi-skilled  Mechanics 

Participants  felt  that  use  of  an  ITI-ALC  system  would  be  easy  (yes)  or  possible  (maybe)  for  a  novice 
technician  (ITI-ALC  Concept  question  5).  When  asked  about  performing  tasks  outside  their  specialty, 
reactions  were  more  hesitant  (ITI-ALC  Concept  question  6).  Responses  indicated  that  they  thought  it 
was  moderately  possible. 

Goal  3.  Perceived  Effect  on  Job  Performance 

In  comparison  to  the  current  TO  system,  the  ITI-ALC  system  was  perceived  to  be  both  faster  and  easier 
along  several  dimensions.  These  dimensions  include  information  retrieval,  form  retrieval,  part  ordering, 
and  other  information  access. 

The  main  concern  in  comparison  with  paper  is  that  searching  a  paper  TO  is  a  fairly  straightforward  task 
(thumb  through  it).  The  ITI-ALC  system’s  current  search  strategies  (organization  of  the  TO  data)  and 
TO  data  access  capabilities  need  improvement.  TO  data  should  be  organized  according  to  aircraft 
location,  technician  specialty,  and  a  numeric  sequence — preferably  all  three  of  these.  For  example,  once 
the  user  logs  in  TO  data  would  be  primarily  available  for  that  technician’s  specialty.  The  technician 
could  limit  searching  based  on  the  aircraft  location  currently  being  worked  (through  graphical  selection 
of  location).  Finally,  TOs  could  be  organized  according  to  numeric  sequence.  Currently  TOs  are 
organized  alphabetically;  therefore  an  alphanumeric  search  strategy  is  not  an  uncommon  concept  to  the 
end-user  population. 
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Goal  4.  Specific  System  Improvements  for  Hardware  and  Software 

Discussion  of  system  improvements  is  organized  according  to  two  main  categories.  Most  of  these 
categories  are  related  to  the  software  and  user  interface  while  the  remaining  are  related  more  specifically 
to  the  hardware. 

Software  and  User  Interface 

Several  topics  concerning  the  ITI-ALC  user  interface  were  evaluated.  These  topics  included  accessing 
information,  entering  information,  viewing  information,  submitting  information,  and  navigating  among 
screens.  Each  of  these  will  be  discussed. 

Access  to  information  using  the  ITI-ALC  system  was  perceived  for  the  most  part  as  faster  and  easier 
than  current  methods.  When  the  system  was  busy  working,  indicators  were  obvious  to  participants  (i.e., 
the  hour-glass  icon  adequate  relayed  that  the  CPU  was  busy). 

Information  entered  into  the  ITI-ALC  system  was  identified  as  easy  to  read  (i.e.,  font  size  was 
sufficient)  and  was  generally  perceived  as  taking  less  time  than  the  current  method  of  data  entry. 

Entering  information  via  the  on-screen  keyboard  was  identified  as  primarily  easy.  Participants  rated 
highly  the  selection  of  part  numbers  through  picture  selection — the  main  critique  was  that  they  wanted 
to  use  this  method  to  select  multiple  parts.  Use  of  the  pen-based  system  to  sketch  and  draw  information 
was  perceived  as  moderately  easy;  however,  these  responses  were  slightly  less  positive.  These  were 
probably  due  to  difficulties  using  the  pen. 

The  pen  posed  many  problems  in  entering  and  selecting  information.  While  handwriting  recognition 
was  generally  good,  participants  had  repeated  problems  selecting  icons,  menu  items,  and  other  screen 
widgets.  These  tasks  were  accomplished  by  “tapping”  or  “double-tapping”  on  the  desired  item  with  the 
pen,  much  like  a  “click”  or  “double-click”  is  used  with  a  mouse.  Many  comments  concerned  the 
sensitivity  of  the  pen.  Perceived  sensitivity  can  be  defined  as  length  of  the  tap,  pressure  of  the  tap,  and 
angle  of  the  pen  during  the  tap.  Clearly,  this  is  an  area  that  requires  improvement  before  a  pen-based 
system  can  be  effectively  implemented  in  the  field. 

Again  for  viewing  information,  participant  felt  that  information  was  easy  to  read — only  the  engineers 
exhibited  any  difficulty,  and  this  problem  could  be  readily  eliminated  with  the  standard-issue  of  17” 
monitors.  Information  presented  on  the  screens  was  perceived  as  adequate  for  accomplishing  the  task 
and  perceived  as  the  same  or  easier  to  read  than  paper. 

Overall,  participants  felt  that  submitting  information  was  easy,  as  was  understanding,  when  the  system 
was  sending  data.  Participants  also  indicated  that  the  ITI-ALC  system  took  less  time  to  submit 
information  than  the  current  method.  However,  initiation  of  a  submission  was  problematic  for 
inspectors.  The  icons  located  at  the  top  left  of  the  screen  (stamp,  envelope  and  +)  caused  several 
problems  in  submitting  information.  Specifically,  icons  in  the  right  menu  bar  were  confused  with  those 
at  the  top,  which  caused  several  user  errors  to  occur. 

While  right  menu  bar  navigation  was  perceived  as  easy  to  understand,  screen  navigation  for  the 
inspectors  was  problematic.  This  problem  was  compounded  by  the  errors  made  submitting  information 
via  the  top  left  icons.  One  example  in  particular  is  upon  sending  the  defects  the  system  navigates  back 
to  the  173  Task  List.  This  automatic  navigation  was  “too  far  back”  and  confusing  to  the  user. 
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System  Hardware 

While  the  ITI-ALC  system  was  perceived  as  comfortable  to  wear,  technicians  indicated  that  they  were 
“worried  about  banging  it  up”  and  that  they  “would  take  it  off  before  performing  maintenance 
inspection.”  These  comments  are  in  accordance  with  their  reactions  that  the  system  interfered  with  the 
task  in  some  cases  and  that  is  was  only  moderately  convenient  while  performing  the  task. 

Straps  were  perceived  as  keeping  the  system  securely  fastened,  although  one  (thin)  technician  had  some 
difficulty  keeping  the  computer  from  sliding  down  into  a  perpendicular  position  to  the  ground. 
Comments  and  question  responses  to  the  pen  have  been  discussed  in  the  previous  section;  however,  it 
should  be  noted  again  that  “tapping  and  touching  caused  problems”  and  that  the  sensitivity  of  pen  needs 
adjustment. 

Reviewing,  selecting,  and  saving  photos  were  perceived  as  fairly  easy.  Additionally,  engineers 
indicated  that  the  digital  camera  would  decrease  the  number  of  trips  to  aircraft.  However,  several 
aspects  of  this  device  require  improvement.  The  digital  camera  provided  adequate  detail  in  some  cases; 
yet,  many  of  the  images  were  out  of  focus  or  did  not  provide  enough  detail.  This  indicates  that 
improvements  need  to  be  made  in  image  quality.  While  sheet  metal  mechanics  did  try  to  capture  in¬ 
focus  images,  this  was  not  an  easy  task.  A  lag  in  the  camera’s  refresh  processing  capability  caused 
further  problems.  As  the  camera  was  pointed  at  an  area  of  interest,  any  slight  movement  would  cause  an 
additional  capture.  However,  the  refresh  rate  on  the  computer  lagged  behind  in  relation  to  the  image 
capture,  resulting  in  a  display  image  that  was  not  always  immediately  representative  of  the  most  recently 
captured  image. 

Recording,  playing  back,  and  reviewing  voice  notes  were  perceived  as  easy  to  use  in  almost  all  cases. 
The  only  concern  involved  potential  interference  from  background  noise  (indicating  the  potential  need 
for  filtering),  yet  engineers  did  not  have  difficulty  understanding  the  voice  notes  attached  to  the  202A 
forms  generated  during  the  test. 

Goal  5.  Lessons  Learned  from  Applications 

Lessons  learned  are  captured  in  the  discussion  of  each  of  the  goals  of  this  field  test. 
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5.  Conclusions 


Lessons  learned  from  the  ITI-ALC  Phase  II  efforts  are  provided  below  and  are  grouped  by  categories 
developed  by  the  team. 


5.1  Domain 

■  Understanding  tasks  of  users  in  detail  is  essential  to  appropriate  applications  design 

■  Processes  for  PDM  appear  generalizable  from  one  product  line  to  another  (same  system  should  work 
with  minor  modifications  for  F-15,  C-130,  C-141,  etc.) 

■  Enabling  linking  to  multiple  information  sources  plus  human  to  human  interaction  only  needed 
occasionally 

■  Intelligent  searching  of  past  work  would  be  helpful  addition  in  the  future. 


5.2  Technical 

■  Handwriting  recognition  needs  to  be  improved  -  use  of  a  better  recognition  package  may  be  essential 
to  technician  acceptance  of  system.  This  is  an  area  that  is  receiving  increased  attention  as  'palm-top' 
computing  becomes  more  widely  used,  and  it  is  reasonable  to  expect  that  better  packages  will  soon 
be  available. 

■  Wireless  coverage  and  stability  may  be  a  challenge  in  the  hangar  environment  due  to  multipath 
interference  that  results  from  the  metal  enclosure  and  the  metal  aircraft  therein.  Permanently 
mounting  antennae  at  elevated  positions  on  the  wall  may  ameliorate  this. 

■  Camera  cord  design  -  we  should  design  and  implement  a  camera  cord  restraint  system  that  doesn’t 
interfere  with  use  of  the  system  when  stowed  and  has  sufficient  flexibility  to  adjust  to  different 
heights/arm  lengths  of  users 


5.3  User  environment 

■  Lighting  glare  and  variable  lighting  in  the  hangars  require  back  lighting  and  active  matrix  displays 
on  the  mobile  computers 

■  Replacement  of  the  mobile  platforms  with  hardened  units  or  improved  cases  to  prevent  damage  from 
dropping/hitting  hard  surfaces  should  be  considered 

■  Allowing  for  flexible/individual  variations,  the  work  processes  (especially  E&I  inspection)  are  more 
viable  with  prototypes 

■  Overall  user  enthusiasm  at  Robins  AFB  for  changing/improving  work  processes  facilitated 
developers’  effort 
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■  Management  support  for  improving  work  processes  and  trying  out  new  technology  a  real  plus  for 
ITI-ALC  Phase  II  team. 

5.4  Commercial  Off  the  Shelf  (COTS)  Technology 

■  Smaller,  lighter  mobile  computing  platforms  are  required  for  extended  use 

■  Lithium  ion  batteries  provide  longer  operation  per  charge,  more  charge/discharge  cycles  before 
replacing  battery 

■  Pentium  120  MHz  mobile  platforms  provided  plenty  of  computing  power  for  thin  client 

■  The  low-cost,  light  weight  digital  ViCam  can  be  made  to  work  adequately  with  careful  use,  but  an 
autofocus  camera  would  provide  more  uniform  results 

■  The  iButton  reader  is  a  cheap,  reliable  electronic  identification  solution  that  the  users  accept  and 
have  confidence  in 

■  Turnkey  applications  for  non-computer  users  should  not  be  built  on  Windows  95;  the  operating 
system  lacks  both  desirable  security  features  and  reliability 

■  Internet  Explorer  4.x  allows  the  underlying  web  application  implementation  to  be  concealed 
(through  its  “full  page’  feature),  but  its  failure  to  use  standard  Windows  widgets  on  web  pages 
reduces  its  compatibility  with  other  COTS  products 

■  Currently  available  voice  recognition  software  can  effectively  support  either  voice  navigation  or 
voice  dictation,  but  not  both 

■  Voice  navigation  software  does  not  currently  work  well  with  web  applications 

■  RF  LAN  networks,  like  wired  networks,  really  achieve  only  50%  or  less  of  the  vendor’s  claimed 
bandwidth 

■  RF  LAN  networks  suffer  from  reduced  bandwidth  when  used  near  operating  microwave  ovens 

■  RF  LAN  network  bandwidth  is  reduced  by  multipath  interference  inside  metal  hangars 

■  RF  LAN  access  points  should  be  positioned  such  that  their  coverage  area  overlap  is  minimal  to 
prevent  mobile  units  from  ‘hunting’  for  the  best  reception  (and  sometimes  missing  transmissions  as  a 
result) 


5.5  Integrated  Product  Team  (IPT)  Structure 

■  Management  of  multiple,  geographically  distributed  subcontractors  consumes  enormous  amounts  of 
program  resources  unless  each  subcontractor  has  clearly  defined  products  to  be  delivered 

■  Contractors  should  have  frequent  contact  with  Lab  management 

■  Experienced  Lab  representatives  should  participate  in  each  IPT  so  that  their  experience  and  expertise 
can  guide  the  contractor  toward  the  Lab’s  desired  goals 
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7.  A CRONYMS  AND  ABBREVIA  TIONS 


This  section  lists  the  acronyms  and/or  abbreviations  used  in  this  document. 


Acronym 

Abbreviation 

A/C 

AFMC 

AFRL 

AFTO 

ALC 

AREP 

BPI 

CAMS 

CDRL 

CIC 

CMU 

COTS 

CSC 

D-level 

DoD 

E&I 

FBE 

FEMS 

FIPS 

FOD 

HESR 

HTML 

ID 

I/F 

I-level 

IMDS 


Definition 

Aircraft 

Air  Force  Materiel  Command 

Air  Force  Research  Laboratory 

Air  Force  Technical  Order 

Air  Logistics  Center 

Aircraft  Repair  Enhancement  Program 

Business  Process  Improvement 

Core  Automated  Maintenance  Improvement 

Contract  Data  Requirements  List 

Communications  Intelligence  Corporation 

Carnegie  Mellon  University 

Commercial-off-the-Shelf 

Computer  Software  Component 

Depot  level 

Department  of  Defense 

Evaluation  and  Inventory 

Field  Based  Evaluation 

Facility  Equipment  Management  System 

Federal  Information  Processing  Standards 

Foreign  Object  Damage 

Logistics  Readiness  Branch 

HyperText  Markup  Language 

Identification 

Interface 

Intennediate  level 

Integrated  Maintenance  Data  System 
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IMIS 

Integrated  Maintenance  Information  System 

I/O 

Input/Output 

IPDF 

Indexed  Portable  Document  Format 

ITI-ALC 

Integrated  Technical  Information  for  the  Air  Logistics  Centers 

LAN 

Local  Area  Network 

LMATL 

Lockheed  Martin  Advanced  Technology  Laboratory 

LMIS 

Lockheed  Martin  Information  Systems 

MHz 

MegaHertz 

O-level 

Organizational  level 

PDM 

Programmed  Depot  Maintenance 

PDMSS 

Programmed  Depot  Maintenance  Scheduling  System 

RAFB 

Robins  Air  Force  Base 

REQ 

Requirement 

RF 

Radio  Frequency 

SEI 

Software  Engineering  Institute 

SOW 

Statement  of  Work 

SSR 

Software  Specification  Review 

SSDD 

System/Segment  Design  Document 

SSS 

System/Segment  Specification 

SW 

Software 

TFT 

Thin-Film  Transistor 

TO 

Technical  Order 

WPAFB 

Wright-Patterson  Air  Force  Base 

WR-ALC 

Warner  Robins-Air  Logistics  Center 
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Appendix  A:  Trade  Studies 


A-1.  Collaboration  Trade  Study 


1.0  Summation 

1.1  Purpose 

The  purpose  of  the  collaboration  trade  study  is  to  investigate  collaborative  software  systems  that 
enable  communication  between  individuals  who  are  not  co-located.  This  communication  takes  several 
forms:  audio,  video  image  transmission,  sharing  a  common  whiteboard,  and  sharing  applications.  The 
intent  is  to  use  the  collaboration  software  to  enable  a  technician  working  at  an  aircraft  to  discuss  a 
problem  and  its  solution  with  an  engineer  at  her  desk  in  a  remote  building.  Audio  will  be  used  to 
discuss  the  problem;  video  to  share  a  picture  of  the  problem;  application  sharing  to  view  and  control 
electronic  technical  manuals;  and  shared  whiteboarding  to  illustrate  ideas  on  both  the  pictures  and 
manuals. 

1.2  Products 

We  evaluated  the  following  products  in  this  trade  study: 


Product  Name 

Intel 

Internet  Video  Phone  2.0  with  Proshare  Technology 

Microsoft 

NetMeeting  2.0 

VocalTec 

InternetPhone  5.0  with  Video 

WhitePine 

CUSeeMe  3.0 

These  products  are  all  software-only  solutions.  They  require  no  proprietary  hardware  such  as  video 
capture  and  compression  boards.  By  making  this  decision  to  evaluate  software-only  solutions,  we 
allow  the  ITI-ALC  project  to  pursue  high  quality  audio  and  video  solutions  independent  of  collaboration 
systems.  In  addition,  this  decision  allows  the  ITI-ALC  project  to  pursue  audio  and  video  hardware 
specifically  tailored  for  each  delivery  platform. 

1.3  Environment 

The  four  products  were  tested  in  an  environment  that  closely  simulates  what  we  expect  to  encounter  in 
the  ITI-ALC  project.  One  test  computer  simulated  a  technician’s  wearable  computer.  This  computer 
was  Carnegie  Mellon  University’s  (CMU)  Tactical  Information  Assistant  Prototype  (TIA-P).  The  TIA-P 
was  built  around  an  Epson  Cardio  card  containing  an  Intel  486-75  Mhz  processor  and  16  MB  of  RAM. 
In  addition,  the  TIA-P  was  equipped  with  an  Aironet  ARLAN  690  two  Mbps  wireless  ethernet  adapter. 
A  Connectix  Quickcam  was  used  for  video  capture,  and  a  Knowles  VR-3185  headset  was  used  for 
audio  capture.  The  TIA-P  was  running  Windows  95. 

The  second  test  computer  simulated  an  engineer’s  desktop  workstation.  This  computer  was  a  dual 
boot  Windows  95/NT  4.0  workstation,  containing  an  Intel  Pentium-133  Mhz  processor  and  32  MB  of 
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RAM.  It  was  directly  connected  to  CMU’s  intranet.  An  Aironet  ARLAN  ethernet  access  point  was 
connected  to  the  intranet  to  supply  connectivity  between  the  TIA-P  and  the  workstation. 

Of  the  four  products,  three  ran  on  both  Windows  NT  4.0  and  Windows  95  platforms.  Intel’s  Internet 
Video  Phone  only  ran  under  Windows  95. 

1.4  Summary  of  Best  Candidate 

While  none  of  the  products  is  ideal,  of  the  four  products  analyzed,  Microsoft  NetMeeting  was  deemed 
preferable:  it  has  the  best  transmission  quality  for  both  audio  and  video,  allows  sharing  of  applications, 
and  subscribes  to  both  the  International  Telecommunication  Union  (ITU)  H.323  audio/video  standards 
and  the  ITU  T.120  whiteboard  standards.  Its  richness  of  features  may  overwhelm  novice  users,  but 
NetMeeting’s  interface  will  be  familiar  to  users  of  Microsoft’s  Internet  Explorer.  A  stable,  commercial 
company  backs  NetMeeting.  Additionally,  NetMeeting  is  the  only  product  we  evaluated  that  exposed 
any  functionality  via  an  application  programming  interface  (API).  This  API,  in  the  form  of  an  ActiveX 
control,  allows  advanced  scripting  of  the  product,  close  integration  of  the  product  with  the  World  Wide 
Web,  and  even  modification  of  the  user  interface.  NetMeeting  brings  together  more  collaborative  tools, 
of  solid  and  dependable  quality,  than  any  other  single  product  on  the  market. 

It  is  interesting  to  note  that  the  September,  1997  issue  of  Byte  magazine  also  reviews  Internet  based 
collaboration  products  in  the  article,  “Lab  Report:  Software:  See  and  Be  Seen  Over  IP”  (Seachrist, 
David;  p.  104).  Byte  selects  NetMeeting  and  WhitePine’s  CUSeeMe  as  the  two  leading  products. 

1.5  Evaluation  of  Other  Products/Tools 

Of  the  remaining  three  products,  only  WhitePine’s  CUSeeMe  also  provides  adequate  tools  for  the 
desired  functionality.  While  there  were  sometimes  significant  disparities  and  delays  in  audio  and  video 
transmission,  audio  quality  was  very  good,  and  video  transmission  quality  was  acceptable.  Whiteboard 
technology  is  supported.  In  fact,  both  CUSeeMe  and  NetMeeting  rely  on  Databeam’s  FarSite 
whiteboarding  technology.  Therefore,  it  is  difficult  to  differentiate  the  two  products  based  on  their 
whiteboards  -  although  we  had  difficulty  setting  up  and  starting  CUSeeMe’s  whiteboard. 

VocalTec’s  InternetPhone  suffered  from  extremely  poor  quality  audio  and  video  transmissions.  Large 
portions  of  spoken  phrases  never  made  it  from  the  wearable  machine  to  the  desktop  speakers.  Video 
resolution  was  good,  but  the  frame  rate  displayed  on  the  desktop  was  quite  poor,  sometimes  taking 
almost  two  seconds  to  register  a  change.  Further,  even  when  the  full  audio  message  got  through,  there 
could  be  as  much  as  a  five  second  lag  between  video  and  audio  reception  for  the  desktop  machine. 

Intel’s  Internet  Video  Phone  was  totally  unacceptable  by  almost  all  measurements.  Audio 
transmissions  were  incomprehensible.  The  video  features  would  not  work  at  all  with  a  black  and  white 
camera,  so  we  were  unable  to  rate  video  transmissions.  There  was  no  ability  to  share  applications  or 
whiteboards.  This  technology  is  in  beta,  and  might  become  more  acceptable  in  the  future,  but  is 
unusable  for  now. 

1.6  Future  Considerations 

The  internet  collaboration  market  is  very  dynamic.  It  is  worthwhile  to  keep  track  of  the  state  of  the 
products  available,  since  the  current  leading  products  and  the  condition  and  responsiveness  of  their 
producing  companies  have  the  potential  for  swift  change. 
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Of  particular  interest  is  PictureTel’s  product  line.  They  offer  two  software-only  products,  LiveShare  Plus 
and  LiveTalk  that  support  application  sharing,  whiteboarding,  and  audio  communication.  Given  their 
experience  in  this  arena,  these  products  should  be  evaluated  against  NetMeeting.  PictureTel’s  video 
conferencing  systems  all  require  proprietary  hardware.  While  proprietary  video  capture  hardware  may 
be  difficult  to  use  in  conjunction  with  a  handheld  computer,  PictureTel’s  support  of  ITU’s  H.323 
standard  would  allow  ITI-ALC  to  use  the  PictureTel  product  at  the  engineer’s  workstation.  ITU’s  H.323 
standard  would  allow  PictureTel’s  video  product  to  interoperate  with  NetMeeting  running  on  a 
technician’s  handheld  computer. 

As  stated  earlier,  DataBeam’s  Farsite  whiteboarding  technology  is  an  established  sector  leader  and 
could  be  pursued  if  only  a  whiteboarding  tool  was  needed. 

The  ITI-ALC  project  is  now  placing  less  emphasis  on  live  video  and  more  on  high-resolution  digital 
picture  stills.  It  may  be  appropriate  to  place  future  focus  on  those  collaboration  products,  which  support 
superior  whiteboarding,  audio,  and  application  sharing  as  opposed  to  products,  which  embody  all 
collaborative  technologies. 

2.0  Trade  Study  Information 

The  products  were  sequentially  installed  and  evaluated.  All  products  were  easily  installed.  However, 
as  noted  before,  CUSeeMe  could  not  locate  where  on  the  wearable  computer  it  had  installed  its 
whiteboard.  Generally,  collaborative  communication  was  initiated  from  the  TIA-P  test  computer.  Both 
audio  and  video  were  transmitted  from  the  TIA-P  to  the  workstation.  The  quality  of  the  audio  and  video 
received  at  the  workstation  was  then  appraised.  Prior  experience  with  audio  and  video  software 
products  (in  particular,  the  products  being  tested)  has  shown  that  the  direction  of  transmission  (TIA-P  to 
workstation  vs.  workstation  to  TIA-P)  does  not  affect  the  quality  of  the  audio  and  video. 

A  test  chart  was  used  to  help  evaluate  the  quality  of  the  transmitted  video.  None  of  the  products 
transmitted  the  chart  in  a  size  and  resolution  sufficient  to  read  the  chart’s  numerical  measurement 
system.  However,  it  was  clear  that  the  NetMeeting  transmitted  video  had  both  the  highest  resolution 
and  best  gray-scale  differentiation. 

3.0  Trade  Study  Assessment  Tables 

The  NetMeeting  (Table  A-1-1)  final  score  (1506)  was  appreciably  higher  (15%)  than  the  CUSeeMe 
(Table  A-1-2)  final  score  (1309).  The  following  list  summarizes  the  key  differentiating  criteria: 

Quality  of  video.  NetMeeting’s  video  frame  rate  was  consistently  two  to  three  times  higher  than 
CUSeeMe’s  frame  rate. 

Timeliness  of  audio  and  video.  NetMeeting’s  audio  arrived  significantly  sooner  than  CUSeeMe’s  audio 
(two  seconds  vs.  five  seconds)  with  no  noticeable  loss  in  quality.  As  a  result,  NetMeeting’s  audio  and 
video  were  substantially  more  synchronized  upon  reception  at  the  remote  workstation. 

Application  Sharing.  NetMeeting  supported  application  sharing  and  collaboration;  CUSeeMe  did  not. 

Application  Programming  Interface.  NetMeeting  exposed  significant  functionality  via  an  ActiveX  control. 
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For  example,  an  included  script  allowed  a  NetMeeting  conference  to  be  launched  and  managed  from  a 
web  page.  While  CUSeeMe  could  also  be  easily  started,  it  offered  no  finer-grained  method  of 
application  control  such  as  starting  and  stopping  video  and  adding  and  removing  users. 

Cost.  NetMeeting  was  free.  CUSeeMe  was  $70,  with  no  support  for  application  sharing. 

Two  products,  VocalTec  InternetPhone  (Table  A-1-3)  and  Intel  Internet  Video  Phone  (Table  A-1-4), 
were  primarily  chosen  as  audio/video  comparison  points.  When  their  audio  and  video  proved  to  be  of 
inferior  quality,  their  trade  studies  were  curtailed. 
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Table  A-1-1  Microsoft  NetMeeting  2.0 


Criteria 


Rqmt  Weigh  Score  Total  Comments 
s  t 


Quality  of  Audio  -  Is  the  audio 
understandable?  Does  the  audio 
breakup,  or  is  the  audio  distorted, 
to  the  point  of  being 
incomprehensible? 


Timeliness  of  Audio  -  Does  the 
audio  arrive  within  two  seconds? 
Faster  is  highly  desirable 


Quality  of  Video  -  Resolution 

For  general  use  needed  to 
establish  context,  160x120 
(pixels)  is  a  minimum.  For 
detailed  shots,  320x240  is  a 
minimum,  while  640x480  is 
preferred  and  may  be  necessary 
depending  upon  the  task. _ 


Quality  of  Video  -  Frame  Rate 

For  general  use  needed  to 
establish  context,  5  frames  per 
second  is  a  minimum.  For 
detailed  shots,  slower  frame  rates 
are  acceptable,  but  this  is  task 
deoendent. 


Quality  and  timeliness  criteria 
are  highly  related  and  can't  be 
judged  independently  of  one 
another.  A  gain  in  one  area 
will  cause  degradation  in 
another.  An  ideal  tool  will 
find  the  best  balance  amongst 
the  criteria  for  the  available 
network  bandwidth. 


Audio  quality  good. 
Occasional  rasping  or  static. 


Quality  of  audio  should  be 
understandable  at  1  sec  delay, 
near  perfect  at  2  sec  delay. 
About  2  second  delay.  _ 


320x240  resolution  possible. 


2-3  frames  per  second.  Note 
that  none  of  the  products 
achieved  5  frames  per  second 
-  this  score  became  a  relative 
score,  comparing  this 
criterium  across  all  products. 
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Table  A-l-1  Microsoft  NetMeeting  2.0  (Continued) 


Quality  of  Video  -  Color  Depth 

For  black  and  white  cameras,  64 
shades  of  gray  need  to  be 
supported.  For  color  cameras,  256 
colors  need  to  be  supported. 

Timeliness  of  Video 

Video  frames  should  be  delayed 
less  than  two  seconds. 

Ease  of  Use 

Minimize  cognitive  load 

Minimize  possibility  of  error 

Learnability 

The  system  should  essentially  be 
"walk  up  and  use"  for  the  end- 
users.  A  maximum  of  5  minutes 
training  to  operate  the  system 
effectively. 

2.0  Reliability 


Not  Applicable 


60 

64  shades  of  gray. 

35 

Previous  video  frames  help  to 
mitigate  video  frame  delays. 
Delay  of  about  1  second. 

70 

Feature  heavy  for  our 
application.  Easy  to  use 
tabbed  dialog  box.  Potential 
for  ActiveX  interface. 

Complexity  leads  to  steeper 
learning  curve,  greater 
potential  for  confusion. 
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Table  A-l 


Multiple  Shared  Application 
Participants 


Sharing  Technology 

Multicast  (excellent),  reflector 
(good),  star  topology  (okay) 


Integratable  Whiteboard  Tool 


Image  Capture  to  Whiteboard 


Multiple  Whiteboard 
Participants 


Whiteboard  Standards 

Adherence  to  International 
Telecommunication  Union  (ITU) 
T.120 


Bandwidth  Adaptable 

Can  the  collaboration  software  be 
optimized  for  bandwidth  available 
on  a  wireless  LAN 
communication  (l-2mb/sec) _ 


Ability  to  Alter  Audio  Quality 


Ability  to  Alter  Audio 
Timeliness 


Ability  to  Alter  Video  Quali 


Ability  to  Alter  Video 
Timeliness 


Audio/Video  Synchronization 

The  closer  the  audio  and  video  are 
kept  synchronized,  the  better.  A 
goal  of  0.5  seconds  is  desirable. 


•1  Microsoft  NetMeeting  2.0  (Continued) _ 


40  Can  more  than  two  users 
share  an  app? 


20  Which  technology  is  used  to 
allow  multiple  shared 
application  and/or  whiteboard 
and/or  audio/video 
participants? 

Star  topology. _ 


0  Can  the  whiteboard  tool  be 
directly  integrated  into  other 
applications?  _ 


Can  video  images  be  captured 
directly  to  the  whiteboard? 
Via  cut  and  paste. 


Can  more  than  two  users 
access  whiteboard 
simultaneously? 


Will  the  whiteboard  tool 
interoperate  with  other 
products? 


56  Can  the  collaboration 

software  adapt  to  available 
bandwidth? 


This  criterion  and  the  next 
three  deal  with  the  ability  to 
trade-off  bandwidth  resources 


6 

3 

18 

6 

2 

12 

6 

8 

48 

6 

8 

48 

8 

8 

64 

Is  the  audio  and  video  kept  in 
synchronization? 

No  more  than  0.5  seconds 
delay _  _ 
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Table  A-l-1  Microsoft  NetMeeting  2.0  (Continued) 


Multiple  Audio/Video 
Participants 


Audio/Video  Standards  - 
Adherence  to  ITU  H.323 


Image  Capture 

Can  a  bitmap  (.BMP),  GIF,  or 
JPEG  image  be  captured? 


Dynamically  Enable/Disable 
Audio /Video 


Dynamically  Control 
Audio/Video  Quality  and 
Timeliness  Tradeoffs 


Can  the  Video  Picture  be 
Zoomed?  (zoom  in/zoom  out) 


Change  Microphone  Capture 
Mode 


Initiate  and  Control 
Collaboration  Session 


API  to  Capture  Images 


API  to  Manipulate  Video 
Window 


Independent  Video  Window 


Add/Remove  Audio/V  ideo 
Codecs 


Filter  Out  Ambient  Audio  Noise 
(echo  cancellation) _ _ 


Cost  per  Seat 
less  than  $100 


Can  more  than  two  users 

videoconference 

simultaneously? 


Will  the  videoconferencing 
tool  interoperate  with  other 
products? _ 


Can  an  image  (video  still)  be 
captured  and  output? 

Via  cut  and  paste.  _ 


Can  the  videoconferencing 
tool  release  the  audio/video 
drivers  so  that  other 
audio/video  tools  can  access 
those  resources? 


Can  quality  and  timeliness 
tradeoffs  be  made  on-the-fly? 


Can  the  microphone  be  set  to 
'open',  'push  to  talk',  or  'voice 
activated'? 


Can  a  collaboration  session  be 
fully  started  without  keyboard 
input  (e.g.,  via  scripts)? _ 


Is  the  video  window 
independent  of  other 
collaboration  windows? 


Can  additional  compression 
components  be  added  at  a 
later  date? 


Can  the  software  filter  out 
noise? 


Free 
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Criteria 


Quality  of  Audio  -  Is  the  audio 
understandable?  Does  the  audio 
breakup,  or  is  the  audio  distorted, 
to  the  point  of  being 
incomprehensible? 


Timeliness  of  Audio  -  Does  the 
audio  arrive  within  two  seconds? 
Faster  is  highly  desirable 


Quality  of  Video  -  Resolution 

For  general  use  needed  to 
establish  context,  160x120 
(pixels)  is  a  minimum.  For 
detailed  shots,  320x240  is  a 
minimum,  while  640x480  is 
preferred  and  may  be  necessary 
depending  upon  die  task. _ 


Quality  of  Video  -  Frame  Rate 

For  general  use  needed  to 
establish  context,  5  frames  per 
second  is  a  minimum.  For 
detailed  shots,  slower  frame  rates 
are  acceptable,  but  this  is  task 
dependent. 


Quality  and  timeliness  criteria 
are  highly  related  and  can't  be 
judged  independently  of  one 
another.  A  gain  in  one  area 
will  cause  degradation  in 
another.  An  ideal  tool  will 
find  the  best  balance  amongst 
the  criteria  for  the  available 
network  bandwidth. 

80 

Audio  quality  good. 

Occasional  rasping  or  static. 

24 

Quality  of  audio  should  be 
understandable  at  1  sec  delay, 
near  perfect  at  2  sec  delay. 

Bad  delays,  up  to  5  seconds. 

70 

320x240  resolution  possible. 

27 

1  frame  per  second 
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Table  A-l-2  WhitePine  CUSeeMe  3.0  (Continued) 


Quality  of  Video  -  Color  Depth 

For  black  and  white  cameras,  64 
shades  of  gray  need  to  be 
supported.  For  color  cameras,  256 

colors  need  to  be  supported. _ 

Timeliness  of  Video 

Video  frames  should  be  delayed 

less  than  two  seconds. 


Ease  of  Use 

Minimize  cognitive  load 
Minimize  possibility  of  error 


Learnability 

The  system  should  essentially 
be  "walk  up  and  use"  for  the 
end-users.  A  maximum  of  5 
minutes  training  to  operate  the 
system  effectively. _ 


2.0  Reliability 


Not  Applicable 


3.0  Maintainability 


Not  Applicable 


64  shades  of  gray  supported 


Previous  video  frames  help  to 
mitigate  video  frame  delays. 
Delay  of  about  1 .5  seconds. 


Relatively  simple,  directory 
appears  on  start  up. 


Simplicity  leads  to  faster 
initial  successful  use. 


4.0  Producibility 

Not  Applicable 

5.0  Capability 


Shared  Application  View 


Shared  Application  Control 


5  0  0  Can  users  view  a  common 

app? 


5  0  0  Can  users  control  a  common 

_ app? _ 
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Table  A-l-2  WhitePine  CUSeeMe  3.0  (Continued) 


Multiple  Shared  Application 
Participants 


Sharing  Technology 

Multicast  (excellent),  reflector 
(good),  star  topology  (okay) 


Integratable  Whiteboard  Tool 


Image  Capture  to  Whiteboard 


Multiple  Whiteboard 
Participants 


Whiteboard  Standards 

Adherence  to  International 
Telecommunication  Union  (ITU) 
T.120 


Bandwidth  Adaptable 

Can  the  collaboration  software  be 
optimized  for  bandwidth  available 
on  a  wireless  LAN 
communication  (l-2mb/sec) _ 


Ability  to  Alter  Audio  Quality 


Ability  to  Alter  Audio 
Timeliness 


Ability  to  Alter  Video  Quali 


Ability  to  Alter  Video 
Timeliness 


Audio/Video  Synchronization 

The  closer  the  audio  and  video  are 
kept  synchronized,  the  better.  A 
goal  of  0.5  seconds  is  desirable. 


Can  more  than  two  users 
share  an  app? 


Which  technology  is  used  to 
allow  multiple  shared 
application  and/or  whiteboard 
and/or  audio/video 
participants? 

Multicast  technology.  _ 


Can  the  whiteboard  tool  be 
directly  integrated  into  other 
applications? _ 


Can  video  images  be  captured 
directly  to  the  whiteboard? 


Can  more  than  two  users 
access  whiteboard 
simultaneously? 


Will  the  whiteboard  tool 
interoperate  with  other 
products? 


Can  the  collaboration 
software  adapt  to  available 
bandwidth? 


This  criterion  and  the  next 
three  deal  with  the  ability  to 
trade-off  bandwidth  resources 


Is  the  audio  and  video  kept  in 
synchronization? 

2-3  second  delay  between 
audio  and  video. 
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Table  A-l-2  WhitePine  CUSeeMe  3.0  (Continued) 


Multiple  Audio/Video 
Participants 

8 

10 

80 

Can  more  than  two  users 

videoconference 

simultaneously? 

Audio/Video  Standards  - 
Adherence  to  ITU  H.323 

5 

10 

50 

Will  the  videoconferencing 
tool  interoperate  with  other 
products? 

Image  Capture 

Can  a  bitmap  (.BMP),  GIF,  or 
JPEG  image  be  captured? 

6 

3 

18 

Can  an  image  (video  still)  be 
captured  and  output? 

Dynamically  Enable/Disable 
Audio/Video 


Dynamically  Control 
Audio/Video  Quality  and 
Timeliness  Tradeoffs 


Can  the  Video  Picture  be 
Zoomed?  (zoom  in/zoom  out) 

Change  Microphone  Capture 
Mode 

Initiate  and  Control 
Collaboration  Session 

IKfffBBliBH 


API  to  Manipulate  Video 
Window 


Independent  Video  Window 


Add/Remove  Audio/Video 
Codecs 


Filter  Out  Ambient  Audio  Noise 
(echo  cancellation) 


Cost  per  Seat 
less  than  $100 


Can  the  videoconferencing 
tool  release  the  audio/video 
drivers  so  that  other 
audio/video  tools  can  access 
those  resources? 


Can  quality  and  timeliness 
tradeoffs  be  made  on-the-fly? 


Can  the  microphone  be  set  to 
'open',  'push  to  talk',  or  'voice 

activated'?  All_ _ 

Can  a  collaboration  session  be 
fully  started  without  keyboard 
input  (e.g.,  via  scripts)? 


Is  the  video  window 
independent  of  other 
collaboration  windows? 


Can  additional  compression 
components  be  added  at  a 
later  date? 


Can  the  software  filter  out 

noise? _ 

About  $70 


Total:  176  1390 
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Table  A-1-3  VocalTec  InternetPhone  5.0 


Criteria 


Rqmt  Weigh  Score  Tota  Comments 
s  t  1 


Quality  of  Audio  -  Is  the  audio 
understandable?  Does  the  audio 
breakup,  or  is  the  audio  distorted, 
to  the  point  of  being 
incomprehensible? 


Timeliness  of  Audio  -  Does  the 
audio  arrive  within  two  seconds? 
Faster  is  highly  desirable 


Quality  of  Video  -  Resolution 

For  general  use  needed  to 
establish  context,  160x120 
(pixels)  is  a  minimum.  For 
detailed  shots,  320x240  is  a 
minimum,  while  640x480  is 
preferred  and  may  be  necessary 
depending  upon  the  task. 


Quality  of  Video  -  Frame  Rate 

For  general  use  needed  to 
establish  context,  5  frames  per 
second  is  a  minimum.  For 
detailed  shots,  slower  frame  rates 
are  acceptable,  but  this  is  task 
dependent.  _ 


Quality  and  timeliness  criteria 
are  highly  related  and  can't  be 
judged  independently  of  one 
another.  A  gain  in  one  area 
will  cause  degradation  in 
another.  An  ideal  tool  will 
find  the  best  balance  amongst 
the  criteria  for  the  available 
network  bandwidth. 


Audio  was  badly  mangled. 
Some  phrases  came  through 
ok,  but  most  were  somewhat  or 
completely  unintelligible. 


Quality  of  audio  should  be 
understandable  at  1  sec  delay, 
near  perfect  at  2  sec  delay. 
About  3  second  delay. 


320x200  available 
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Table  A-l-3  VocalTec  InternetPhone  5.0  (Continued) 


Quality  of  Video  -  Color  Depth  6  10  60  64  shades  of  gray  supported 

For  black  and  white  cameras,  64 
shades  of  gray  need  to  be 
supported.  For  color  cameras,  256 

colors  need  to  be  supported. _ 

Timeliness  of  Video  5  3  15  Previous  video  frames  help  to 

Video  frames  should  be  delayed  mitigate  video  frame  delays. 

less  than  two  seconds. _ 

Ease  of  Use  10  4  40  Poor  performance  on  basic 

Minimize  cognitive  load  functionality 

Minimize  possibility  of  error _ 

Learnability  10  7  70 

The  system  should  essentially 

be  "walk  up  and  use"  for  the 

end-users.  A  maximum  of  5 

minutes  training  to  operate  the 

system  effectively. _ 


5.0  Capability 


Shared  Application  View 

5 

0 

Can  users  view  a  common 
app? 

Shared  Application  Control 

5 

0 

0 

Can  users  control  a  common 
app? 

Multiple  Shared  Application 
Participants 

4 

0 

0 

Can  more  than  two  users  share 
an  app? 
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Table  A-l-3  VocalTec  InternetPhone  5.0  (Continued) 


Sharing  Technology 

Multicast  (excellent),  reflector 
(good),  star  topology  (okay) 


Integratable  Whiteboard  Tool 


Image  Capture  to  Whiteboard 


Multiple  Whiteboard 
Participants 


Whiteboard  Standards 

Adherence  to  International 
Telecommunication  Union  (ITU) 
T.120 


Bandwidth  Adaptable 

Can  the  collaboration  software  be 
optimized  for  bandwidth  available 
on  a  wireless  LAN 
communication  (l-2mb/sec) _ 


Ability  to  Alter  Audio  Quality 


Ability  to  Alter  Audio 
Timeliness 


Ability  to  Alter  Video  Quali 


Ability  to  Alter  Video 
Timeliness 


Audio/Video  Synchronization 

The  closer  the  audio  and  video  are 
kept  synchronized,  the  better.  A 
goal  of  0.5  seconds  is  desirable. 


Which  technology  is  used  to 
allow  multiple  shared 
application  and/or  whiteboard 
and/or  audio/video 
participants? 

Star  topology  used. 


Can  the  whiteboard  tool  be 
directly  integrated  into  other 
applications? 


Can  video  images  be  captured 
directly  to  the  whiteboard? 


Can  more  than  two  users 
access  whiteboard 
simultaneously? _ 


Will  the  whiteboard  tool 
interoperate  with  other 
products? 


Can  the  collaboration  software 
adapt  to  available  bandwidth? 


This  criterion  and  the  next 
three  deal  with  the  ability  to 
trade-off  bandwidth  resources 


Is  the  audio  and  video  kept  in 
synchronization? 

A  disparity  of  as  much  as  5 
seconds  was  noticed. 
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Table  A-l-3  VocalTec  InternetPhone  5.0  (Continued) 


Multiple  Audio/Video 
Participants 


Audio/Video  Standards  - 
Adherence  to  ITU  H.323 


Image  Capture 

Can  a  bitmap  (.BMP),  GIF,  or 
JPEG  image  be  captured? 


Dynamically  Enable/Disable 
Audio/Video 


Independent  Video  Window 


Dynamically  Control 
Audio/Video  Quality  and 
Timeliness  Tradeoffs 

Can  the  Video  Picture  be 
Zoomed?  (zoom  in/zoom  out) 

Change  Microphone  Capture 
Mode 

Initiate  and  Control 
Collaboration  Session 

API  to  Capture  Images 

API  to  Manipulate  Video 
Window 

i 

Add/Remove  Audio/Video 

Codecs 

Filter  Out  Ambient  Audio  Noise 
(echo  cancellation) 

Cost  per  Seat 
less  than  $100 

80 

Can  more  than  two  users 

videoconference 

simultaneously? 

50 

Will  the  videoconferencing 
tool  interoperate  with  other 
products? 

36 

Can  an  image  (video  still)  be 
captured  and  output? 

0 

Can  the  videoconferencing 
tool  release  the  audio/video 
drivers  so  that  other 
audio/video  tools  can  access 
those  resources? 

0 

Can  quality  and  timeliness 
tradeoffs  be  made  on-the-fly? 

56 

48 

Can  the  microphone  be  set  to 
'open',  'push  to  talk',  or  'voice 
activated'? 

0 

Can  a  collaboration  session  be 
fully  started  without  keyboard 
input  (e.g.,  via  scripts)? 

0 

Is  the  video  window 
independent  of  other 
collaboration  windows? 


8 

36 

Can  additional  compression 
components  be  added  at  a 
later  date? 

6 

24 

Can  the  software  filter  out 
noise? 

5 

40 

About  $50 
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Table  A-1-4  Intel  Internet  Video  Phone  2.0 


Criteria 


Quality  of  Audio  -  Is  the  audio 
understandable?  Does  the  audio 
breakup,  or  is  the  audio  distorted, 
to  the  point  of  being 
incomprehensible? 


Timeliness  of  Audio  -  Does  the 
audio  arrive  within  two 
seconds?  DFaster  is  highly 
desirable 


Quality  of  Video  -  Resolution 

For  general  use  needed  to 
establish  context,  160x120 
(pixels)  is  a  minimum.  For 
detailed  shots,  320x240  is  a 
minimum,  while  640x480  is 
preferred  and  may  be  necessary 
depending  upon  the  task.  _ 


Quality  of  Video  -  Frame  Rate 

For  general  use  needed  to 
establish  context,  5  frames  per 
second  is  a  minimum.  For 
detailed  shots,  slower  frame  rates 
are  acceptable,  but  this  is  task 
dependent.  _ 


Tota  Comments 

I 


Quality  and  timeliness  criteria 
are  highly  related  and  can't  be 
judged  independently  of  one 
another.  A  gain  in  one  area 
will  cause  degradation  in 
another.  An  ideal  tool  will 
find  the  best  balance  amongst 
the  criteria  for  the  available 
network  bandwidth. 


Audio  badly  mangled.  Hard  to 
get  a  single  word  though 
clearly.  Many  phrases 
completely  disappear. 


Quality  of  audio  should  be 
understandable  at  1  sec  delay, 
near  perfect  at  2  sec  delay. 
About  3  seconds  delay. 


Video  would  not  transmit. 


0  Video  would  not  transmit. 
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Table  A-l-4  Intel  Internet  Video  Phone  2.0  (Continued) 


Quality  of  Video  -  Color  Depth 

For  black  and  white  cameras,  64 
shades  of  gray  need  to  be 
supported.  For  color  cameras,  256 
colors  need  to  be  supported. 

6 

0 

0 

Video  would  not  transmit. 

Timeliness  of  Video 

Video  frames  should  be  delayed 
less  than  two  seconds. 

5 

0 

0 

Previous  video  frames  help  to 
mitigate  video  frame  delays. 

Ease  of  Use 

Minimize  cognitive  load 

Minimize  possibility  of  error 

10 

30 

Poor  performance  in  basic 
functionality 

Leamability 

The  system  should  essentially 
be  "walk  up  and  use"  for  the 
end-users.  A  maximum  of  5 
minutes  training  to  operate  the 
system  effectively. 

10 

6 

60 

5.0  Capability 

Shared  Application  View  5  0  0  Can  users  view  a  common 

_ app? _ 

Shared  Application  Control  5  0  0  Can  users  control  a  common 

_ app? _ 

Multiple  Shared  Application  4  0  0  Can  more  than  two  users  share 

Participants _ an  app? _ 
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Table  A-1-4  Intel  Internet  Video  Phone  2.0  (Continued) 


Sharing  Technology 

Multicast  (excellent),  reflector 
(good),  star  topology  (okay) 


Integratable  Whiteboard  Tool 


Image  Capture  to  Whiteboard 


Multiple  Whiteboard 
Participants 


Whiteboard  Standards 

Adherence  to  International 
Telecommunication  Union  (ITU) 
T.120 


Bandwidth  Adaptable 

Can  the  collaboration  software  be 
optimized  for  bandwidth  available 
on  a  wireless  LAN 
communication  (l-2mb/sec) _ 


Ability  to  Alter  Audio  Quality 


Ability  to  Alter  Audio 
Timeliness  _ 


Ability  to  Alter  Video  Quality 


Ability  to  Alter  Video 
Timeliness 


Audio/Video  Synchronization 

The  closer  the  audio  and  video  are 
kept  synchronized,  the  better.  A 
goal  of  0.5  seconds  is  desirable. 


Which  technology  is  used  to 
allow  multiple  shared 
application  and/or  whiteboard 
and/or  audio/video 
participants? 

Reflector  technology  used. 


Can  the  whiteboard  tool  be 
directly  integrated  into  other 
applications? _ 


Can  video  images  be  captured 
directly  to  the  whiteboard? 


Can  more  than  two  users 
access  whiteboard 
simultaneously?  _ 


Will  the  whiteboard  tool 
interoperate  with  other 
products? 


Can  the  collaboration  software 
adapt  to  available  bandwidth? 


This  criterion  and  the  next 
three  deal  with  the  ability  to 
trade-off  bandwidth  resources 


Is  the  audio  and  video  kept  in 
synchronization? 

No  video  images  available  for 
comparison. 
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Table  A-l-4  Intel  Internet  Video  Phone  2. 


Multiple  Audio/Video 
Participants 


Audio/Video  Standards  - 
Adherence  to  ITU  H.323 


Image  Capture 

Can  a  bitmap  (.BMP),  GIF,  or 
JPEG  image  be  captured? 


Dynamically  Enable/Disable 
Audio/Video 


Dynamically  Control 
Audio/Video  Quality  and 
Timeliness  Tradeoffs 


Can  the  Video  Picture  be 
Zoomed?  (zoom  in/zoom  out) 


Change  Microphone  Capture 
Mode 


Initiate  and  Control 
Collaboration  Session 


API  to  Capture  Images 


API  to  Manipulate  Video 
Window 


Independent  Video  Window 


Add/Remove  Audio/V  ideo 
Codecs 


Filter  Out  Ambient  Audio  Noise 
(echo  cancellation 


Cost  per  Seat 

less  than  $100 


0 

Can  more  than  two  users 

videoconference 

simultaneously? 

50 

Will  the  videoconferencing 
tool  interoperate  with  other 
products? 

0 

Can  an  image  (video  still)  be 
captured  and  output? 

0 

Can  the  videoconferencing 
tool  release  the  audio/video 
drivers  so  that  other 
audio/video  tools  can  access 
those  resources? 

16 

Can  quality  and  timeliness 
tradeoffs  be  made  on-the-fly? 

80 

Can  the  microphone  be  set  to 
'open',  'push  to  talk',  or  'voice 
activated'? 

0 

Can  a  collaboration  session  be 
fully  started  without  keyboard 
input  (e.g.,  via  scripts)? 

0 

0 

0 

Is  the  video  window 
independent  of  other 
collaboration  windows? 

0 

Can  additional  compression 
components  be  added  at  a  later 
date? 

20 

Can  the  software  filter  out 
noise? 

80 

Free 
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4.0  Product  Experience 

As  an  all-in-one  solution,  NetMeeting  cannot  be  beaten.  It  is  the  only  software  system  the  authors  are 
aware  of  that  offers  the  full  suite  of  collaborative  technologies.  Other  products  may  excel  at  individual 
technologies.  Often  times,  this  excellence  comes  at  the  expense  of  dedicated  hardware  or  non- 
standards-based  algorithms.  For  example,  CUSeeMe  version  2.1.1  did  not  support  the  ITU  H.323 
audio/video  standard.  The  authors  have  yet  to  find  a  product  with  the  quality  of  that  version’s  audio  and 
video.  Standards  compliance  often  comes  with  performance  penalties. 

5.0  Source  Summary 


Product  Name  /  Contact 

Intel 

Internet  Video  Phone  2.0  with  Proshare  Technology 
http://connectedpc.com/iaweb/cpc/iivphone/index.htm 

Microsoft 

NetMeeting  2.0 

http://www.microsoft.com/netmeeting/ 

VocalTec 

InternetPhone  5.0  with  Video 

http://www.vocaltec.com/products/iphone5/index.html 

WhitePine 

CUSeeMe  3.0 

http://www.  cuseeme.com/cu30-win.  htm  1 

DataBeam 

FarSite 

http://www.databeam.com/Products/FarSite/ 

PictureTel 

LiveShare  Plus 

http://www.oicturetel.com/liveshrp.htm 
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A-2.  Database  Management  Systems  Trade  Study 


1.0  Summation 

1.1  Purpose 

The  purpose  of  the  Database  Management  Systems  (DBMSs)  trade  study  is  to  investigate  and 
evaluate  the  current  state  of  the  art  of  DBMSs  and  recommend  the  best  candidate  that  fulfills  the 
requirements  for  the  ITI-ALC  Program  Phase  II.  The  primary  requirement  for  the  DBMS  is  to  support 
the  Computerized  Maintenance  Management  System  (CMMS)  chosen  by  the  ITI-ALC  team. 
Secondarily  the  DBMS  will  be  utilized  to  support  most  ITI-ALC  Program  Phase  II  functionality  that 
requires  persistent  data  storage,  such  as  but  not  limited  to  the  following: 

•  Mechanics  real-time  annotation  (both  written  and  spoken) 

•  Three-dimensional  models  such  as  VRML 

•  Diagnostic  information 

•  Streaming  audio  and  video 

•  Electronic  technical  manuals 

•  Digitized  still  images 

•  Illustrations,  graphics,  and  schematics 

1.2  Products 

For  the  purpose  of  this  trade  study  a  Database  Management  System  will  be  defined  as  and  have  the 
following  characteristics: 

•  A  set  of  software  packages  (tools)  and  servers  (processes)  that  manage  persistent  data  stores 

•  Support  for  simple  datatypes  such  as  integers,  scientific  floating-point,  character  strings, 
date/time,  and  money 

•  Extendable  to  handle  "rich”  datatypes  or  "objects"  that  represent  complex  internal  structures, 
attributes,  and  behavior  and  that  support  new  search  methods 

•  Intrinsically  “aware”  of  new  datatypes 

•  Support  for  queries  of  both  simple  and  complex  datatypes 

•  Supported  by  leading  CMMS  products 

The  methodology  employed  to  downselect  the  DBMS  products  started  by  considering  the  available 
DBMS  technologies.  Candidate  DBMS  technologies  were  analyzed  to  determine  which  DBMS 
technology  best  meets  the  program’s  needs.  The  initial  DBMSs  technology  candidates  considered 
included  Relational  Database  Management  System  (RDBMS),  Object-Oriented  Database  Management 
System  (OODBMS),  and  Object/Relational  Database  Management  System  (O/RDBMS).  The 
candidate  DBMS  technology  that  best  represented  the  definition  of  this  trade  study  was  O/RDBMS 
technology. 

Candidate  O/RDBMS  products  were  gathered,  analyzed,  and  downselected  based  on  CMMS  support, 
technical  quality,  maturity,  and  market  share.  IBM  Universal  Database  was  initially  considered  for 
evaluation  but  was  not  downselected  because  none  of  the  CMMSs  under  consideration  worked  with  the 
product.  The  following  products  where  evaluated  for  the  DBMSs  trade  study: 
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Company: 

Version 

Informix  Inc. 

INFORMIX-Universal  Server 

v9.11  Beta 

Oracle  Corp. 

ORACLE8  Universal  Server 

V8.0.3 

Adaptive  Server 

vll.5  Beta 

1.3  Environment 

The  computer  used  to  evaluate  the  three  DBMS  products  was  a  Dell  Windows  NT  4.0  workstation, 
containing  an  Intel  Pentium-133  MHz  processor,  96  MB  of  RAM,  and  8.4  GB  of  disk. 

1.4  Summary  of  Best  Candidate 

Out  of  the  three  leading  DBMS  technologies  reviewed,  O/RDBMS  technology  was  deemed  to  best  fit 
the  ITI-ALC  program’s  requirements,  for  both  demonstration  and  production.  O/RDBMS  technology 
was  selected  due  to  O/RDBMSs  strong  querying  capabilities  of  complex  datatypes. 

O/RDBMSs  are  able  to  intelligently  manage  complex  data  not  as  binary  large  objects  (BLOBs)  but 
natively  as  objects.  This  means  very  high  performance  query  and  transaction  access  to  rich  complex 
data.  O/RDBMSs  provide  an  SQL  interface  for  rapid  application  development  and  maintenance,  as 
well  as  delivering  scalability  and  manageability.  O/RDBMSs  allow  you  to  define  both  the  structure  and 
behavior  of  data  needed  to  define  particular  applications.  O/RDBMS  servers  can  make  intelligent 
choices  to  optimize  access  to  the  complex  data,  and  users  can  define  functions  that  manipulate  the 
data  right  in  the  server  itself,  or  take  advantage  of  datatype  and  function  extensions  written  by  others. 

The  three-downselected  O/RDBMS  products  all  employ  different  approaches  to  implementing  an 
O/RDBMS: 

•  Sybase’s  approach  employs  a  separate  servers  single  operational  model.  In  Sybase  separate 
servers  single  operational  model  approach  relational  and  specialty  datatypes  are  stored  in 
separate  data  stores.  Separate  servers  are  held  together  by  a  glue  layer,  which  is  managed 
and  monitored  by  common  services. 

•  Oracle’s  approach  employs  middleware  simulation  (object  simulator).  The  object  simulator 
simulates  specialized  object-relational  functionality  outside  the  DBMS  in  a  middleware  layer. 
This  approach  involves  little  RDBMS  changes;  rather,  it  puts  a  specialized  wrapper  or  simulator 
around  the  DBMS. 

•  Informix  employs  a  hybrid  (object  top)  approach.  In  the  object  top  approach  Informix  rewrote 
the  "top  half'  of  its  relational  engine  to  support  object-relational  functionality,  including 
extensible,  complex  datatypes.  This  approach  involved  building  the  upper  half  of  an  object- 
relational  engine  and  then  layering  it  onto  the  storage  manager  and  transaction  system  of  a  their 
RDBMS. 

Sybase's  Adaptive  Server  was  not  selected  due  to  low  trade  study  assessment  score  and  deficiencies 
in  defining  complex  datatypes,  as  reflected  in  the  Trade  Study  Assessment  Table.  Sybase's  Adaptive 
Server  extendable  O/RDBMS  approach  is  less  desirable  when  compared  to  the  other  two  approaches. 
Sybase's  Adaptive  Server  is  less  desirable  because  of  the  inability  for  the  user  to  easily  extend  the 
DBMS  if  at  all.  Also  since  Sybase's  Adaptive  Server  architecture  has  loosely  coupled  servers  a 
relational  engine  and  specialty  data  engines  all  held  together  by  a  glue  layer  data  is  stored  in  different 
servers  depending  on  their  datatype.  Therefore,  this  object-simulation  environment  will  require  cross¬ 
server  joins  for  queries  that  select  data  from  multiple  servers,  which  should  cause  costly  performance 
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hits  due  to  server  coordination  coordinated  by  the  glue  layer. 

However,  Sybase's  Adaptive  Server  comes  with  a  comprehensive  set  of  tools  that  seemed  to  be  both 
solid  and  dependable.  Sybase's  technical  support  system  seems  to  be  a  little  less  then  adequate  with 
an  average  four  hours  turn  around  time  after  call  is  placed.  Sybase's  Adaptive  Server  price  per 
concurrent  seat  is  good  but  the  maintenance  structure  is  less  then  desirable.  For  these,  reason 
Sybase's  Adaptive  Server  is  deemed  inadequate  for  both  the  ITI-ALC  program  demonstration  and 
production. 

Informix's  INFORMIX-Universal  Server  had  the  second  highest  Trade  Study  Assessment  Score,  which 
only  trailed  Oracle's  ORACLE8  Universal  Server  score  by  sixteen  raw  points  and  one  hundred  and 
fifteen  weighted  points.  Informix's  INFORMIX-Universal  Server  extendable  O/RDBMS  approach  is  the 
most  desirable  when  compared  to  the  other  two  approaches.  The  reasoning  behind  the  preference  for 
the  INFORMIX-Universal  Server  extendable  O/RDBMS  approach  is  listed  below: 

•  Informix  tightly  integrates  new  datatypes  with  the  DBMS  server  through  its  DataBlade  API;  Oracle 
loosely  couples  with  new  datatypes  through  CORBA. 

•  The  notion  that  the  DBMS  server  is  intrinsically  aware  of  new  datatypes  in  a  fashion  similar  to 
native  RDBMS  datatypes  such  as  integer. 

•  The  extendibility  of  the  indexing  and  optimization  system  due  to  the  DBMS  server  awareness  of 
new  datatypes. 

•  The  theoretical  performance  difference  between  extending  O/RDBMS  server  through  a  fast  server 
based  local  API  versus  a  network  based  API. 

Informix’s  INFORMIX-Universal  Server  appears  to  be  a  technically  competent  package.  However, 
Informix  failed  to  provide  the  INFORMIX-Universal  Server  in  time  for  a  lengthy  evaluation  and  to 
provide  adequate  technical  support  for  the  evaluation.  In  their  defense,  Informix  did  warn  about 
difficulties  of  getting  technical  support  so  late  in  the  Beta  cycle.  Informix's  INFORMIX-Universal  Server 
for  the  Windows  NT  platform  is  expected  to  be  generally  available  in  December  1997.  Informix's 
INFORMIX-Universal  Server  lacked  server  side  CORBA  support  and  Lockheed  Martin  Information 
Systems  has  very  little  to  no  experience  with  Informix  products.  Informix  INFORMIX-Universal  Server 
comes  with  a  comprehensive  set  of  tools  but  the  trade  study  assessor  had  difficulties  in  setup  with 
some  of  the  tools  and  some  were  just  not  available  for  evaluation. 

In  addition,  a  stable  company  does  not  back  Informix’s  INFORMIX-Universal  Server  currently  even 
though  Informix  enjoys  the  second  largest  market  share  in  the  DBMS  industry.  Informix's  INFORMIX- 
Universal  Server  price  per  concurrent  seat  and  maintenance  structure  is  higher  than  Oracle’s  without 
additional  price  negotiations.  For  these  reasons  Informix's  INFORMIX-Universal  Server  is  not  deemed 
preferable  at  this  time. 

Of  the  three  O/RDBMS  products  evaluated,  Oracle  ORACLE8  Universal  Server  was  deemed  most 
preferable.  Oracle  ORACLE8  Universal  Server  had  the  highest  Trade  Study  Assessment  Score,  has  a 
CORBA  based  open  architecture,  and  Lockheed  Martin  Information  Systems  company  has  over  seven 
years  of  experience  with  Oracle  products.  While  the  Oracle  ORACLE8  Universal  Server  extendable 
O/RDBMS  approach  is  less  desirable  than  Informix  O/RDBMS’s  approach,  it  is  still  capable  of 
supporting  ITI-ALC  program  needs.  A  stable  company  backs  ORACLE8  Universal  Server  and  Oracle 
enjoys  the  largest  market  share  in  the  DBMS  industry.  In  addition,  Oracle  ORACLE8  Universal  Server 
comes  with  a  comprehensive  set  of  tools  that  are  both  solid  and  dependable.  Oracle  has  also  built  a 
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strong  technical  support  system.  Oracle  ORACLE8  Universal  Server  price  per  concurrent  seat  and 
maintenance  structure  is  also  appealing.  For  these,  reason  Oracle  ORACLE8  Universal  Server  is 
deemed  preferable  for  both  the  ITI-ALC  program  demonstration  and  production. 

1.5  Evaluation  of  Other  Products/Tools 

The  two  DBMS  candidate  technologies  not  selected  each  have  their  own  strengths  that  make  them 
appropriate  for  particular  classes  of  data  management  problems.  However,  neither  one  completely 
meets  the  DBMS  criteria  set  forth  by  this  trade  study. 

RDBMSs  provide  high-speed,  short-running  queries  and  transactions  on  simple  data.  A  modern 
RDBMS  is  scalable,  robust,  and  manageable,  and  provides  access  to  data  by  content,  using  the  SQL 
language  for  phrasing  queries.  It  is  an  excellent  platform  for  developing  applications  for  both  high¬ 
speed  transactions  and  flexible  decision  support  on  its  supported  datatypes. 

However,  existing  RDBMSs  provide  only  limited  support  for  complex  data,  which  they  store  as  BLOBs 
that  cannot  be  indexed,  searched,  or  manipulated  within  the  server. 

OODBMSs  provide  persistent  storage  for  complex  objects  manipulated  by  object-oriented  (00) 
programming  languages  like  C++  and  SmallTalk.  Applications  written  in  these  languages  expect  to 
follow  pointers  to  locate  data  and  rely  on  the  OODBMS  to  support  a  large  virtual  memory  space. 
Because  the  fundamental  way  to  navigate  through  an  OODBMS  is  by  pointer  following,  provision  of  a 
highly  optimized  query  language  has  not  been  a  major  concern  for  OODBMS  vendors.  The  OODBMSs 
currently  available  do  not  scale  well  either  over  very  large  data  stores  or  very  large  numbers  of  users,  a 
problem  handled  well  by  some  of  the  RDBMS  products. 

Typical  applications  of  an  OODBMS  are  for  a  small  number  of  users  to  perform  complex  manipulations 
of  object  collections  in  the  tens  of  megabytes  range.  Examples  include  the  manipulation  of  CAD 
(computer-aided  design)  drawings  and  other  complex,  structured  data. 

The  two  DBMS  technologies  discussed  above  fail  to  provide  either  complex  data  support  or  highly 
scalable  and  robust  query  access  to  large  data  stores. 

1.6  Future  Considerations 

The  Database  Management  System  market  place  is  very  dynamic.  It  is  worthwhile  to  keep  track  of  the 
state  of  the  products  available,  since  the  currently  leading  products  and  the  condition  and 
responsiveness  of  their  producing  companies  have  the  potential  for  swift  change. 
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2.0  Trade  Study  Information 

2.1  Informix  INFORMIX-Universal  Server  Architecture  Summary 

INFORMIX-Universal  Server  (IUS)  (Table  A-2-1),  O/RDBMS  architecture  is  based  on  two  technologies: 
Informix  Dynamic  Scalable  Architecture  (DSA)  and  lllustra  DataBlade  technology.  Dynamic  Scalable 
Architecture  provides  scalability  with  parallel  processing  at  the  core  of  its  architecture.  INFORMIX- 
Universal  Server  can  be  extended  to  manage  new  kinds  of  data  by  means  of  DataBlade  modules. 
DataBlade  modules  define  new  data  structures,  new  functions  that  manipulate  them,  and  optionally 
new  access  methods  to  provide  fast  access  to  the  data.  A  DataBlade  module  enables  the  database 
server  to  provide  the  same  level  of  support  for  new  data  types  that  it  provides  for  built-in  data  types. 
DataBlade  modules  can  be  thought  of  as  an  object-oriented  package,  similar  to  a  C++  class  that 
encapsulates  specialized  data  types,  such  as  images. 

The  important  architectural  components  of  Dynamic  Scalable  Architecture  and  DataBlade  technology 
are: 

•  Dynamic  Scalable  Architecture  -  provides  high  scalability  with  parallel  processing  and  the  ability 
to  dynamically  adjust  database  parameters 

•  "Pluggable"  objects  called  DataBlade  modules  that  are  manageable  and  provide  extensible 
functionality,  including  geo-spatial  data,  time  series  data,  multi-media,  and  image  content  and 
text. 

•  DataBlade  API  -  standardizes  interfaces  that  enable  server  extension 

•  Integrated  development  and  management  of  DataBlade  modules. 
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2.2  Oracle  ORACLE8  Universal  Server  Architecture  Summary 

0RACLE8  Universal  Server  (Table  A-2-2),  O/RDBMS  architecture  is  based  on  Oracle's  Network 
Computing  Architecture  (NCA).  Network  Computing  Architecture  is  based  on  open  technologies  and 
standards:  At  the  core  of  Network  Computing  Architecture  are  the  open  and  de  facto  standards: 
HTTP/HTML,  CORBA  2.0  and  Java. 

The  Network  Computing  Architecture  combines  the  Web  technologies  of  HTTP  and  HTML  with  the 
object  technologies  of  CORBA  2.0  to  form  the  basis  for  a  distributed  computing  environment.  CORBA 
provides  Network  Computing  Architecture  with  a  distributed-object  environment,  which  includes  HOP  for 
object  interoperability  and  IDL  for  language-neutral  interfaces.  The  Network  Computing  Architecture 
uses  Java  to  provide  extensibility,  portability,  and  a  de  facto  programming  language  throughout  the 
architecture  but  Java  is  not  available  (server  side)  in  the  first  release.  The  Network  Computing 
Architecture  also  supports  ActiveX/COM  clients  through  open  COM/CORBA  interoperability 
specifications. 

The  important  architectural  components  of  Network  Computing  Architecture  are: 

•  "Pluggable"  objects  called  cartridges  that  are  manageable  and  provide  extensible  functionality, 
which  includes  support  for  geo-spatial  data,  time  series  data,  multi-media,  and  image  content 
and  text. 

•  Open  protocols  and  standardized  interfaces  that  enable  communication  among  cartridges 
through  a  software  bus  called  Inter-Cartridge  Exchange  (ICX). 

•  Extensible  clients,  application  servers,  and  database  servers. 

•  A  family  of  clients: 

•  Oracle’s  Web  Application  Server 

•  Oracle’s  universal  data  server 

•  Integrated  development  and  management  of  cartridges. 
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2.3  Sybase  Adaptive  Server  Architecture  Summary 

Adaptive  Server  (Table  A-2-3),  O/RDBMS  architecture  is  one  part  of  the  Adaptive  Component 
Architecture.  The  Adaptive  Component  Architecture  provides  a  single  operational  model  (management 
services)  for  middle-tier  servers  as  well  as  data  servers.  The  Adaptive  Server  Architecture  brings  the 
existing  Sybase  database  products  into  a  unified  architecture  with: 

•  Optimized  data  stores  and  access  methods 

•  Single  programming  model  across  each  of  data  stores 

•  Single  operational  model  across  stores 

•  Specialty  datatypes  in  their  own  data  stores 

•  Transaction-based  processing  across  each  of  these  data  stores 

The  Sybase  Adaptive  Server  strategy  is  based  on  the  hypothesis  that  some  features  of  a  data  server 
need  to  be  optimized  for  specific  purposes,  while  other  features  need  to  be  common  across  all 
configurations.  Adaptive  Server  provides  separate  components  for  each  of  the  server  tasks.  In  this 
way,  Sybase  states  optimized  components  for  data  access  can  coexist  with  a  Common  Language 
Component  and  other  components  for  accessing  specialty  data  types. 

The  important  architectural  components  of  Adaptive  Component  Architecture  are: 

•  Common  Language  Processor  -  provides  a  consistent  language,  interface  across  all  data 
stores. 

•  Component  Integration  Layer  -  provides  access  to  specialty  datatypes,  relational  data  stored  in 
other  vendors'  databases,  and  distributed  data  access. 

•  Optimized  relational  data  stores  -  Data  access  and  storage  are  linked.  Indexing  systems, 
management  requirements,  and  other  aspects  of  data  storage  are  features  of  the  data  store  (the 
data  access  engine  and  associated  physical  store). 

•  Specialty  datatype  stores  -  Specialty  datatypes,  including  geo-spatial  data,  time  series  data, 
multi-media,  image  content  and  text,  reside  in  their  own  data  stores. 

•  Single  operational  model  -  Management  and  monitoring  services  will  be  shared  across  all 
stores,  to  enable  system-wide  administration. 
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3.0  Trade  Study  Assessment  Tables 

3.1  Informix  Trade  Study  Assessment  Tables 


Table  A-2-1  Informix  INFORMIX-Universal  Server  for  Window  NT  v9. 11  Beta 

CRITERIA 

RQMTS 

** 

M 

HH 

O 

35 

SCORE 

TOTAL 

COMMENTS 

1.0  Performance 

1.1  Cost-based,  syntax- 
independent 
optimization 

6 

8 

48 

Statistics  are  kept  current 
by  manually  issuing 
analyzing  schema 
command(s).  Query  plan 
will  be  affected  if  statistics 
are  too  far  out  of  date. 

Table-driven  extensible 
query  optimizer 

1.2  Shared  database 
buffer,  data 
dictionary,  SQL 
statements,  and  stored 
procedures  cache 

6 

6 

36 

1.3  B-tree  single  and 

concatenated  column 
indexes 

6 

8 

48 

Extensible  indexing  system 
through  DataBlades 
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Table  A-2-1  Informix  INFORMIX-Universal  Server  for  Window  NT  v9. 11  Beta 

(Continued) 

1.4  Contention-free,  non- 
blocking,  multi¬ 
version,  and  read- 
consistent  queries 

10 

6 

60 

By  default,  each  insert, 
update,  and  delete 
statement  is  considered  a 
single  transaction. 

For  contention-free,  non- 
blocking,  multi-version, 
and  read-consistent  queries 
you  have  to  explicitly 
define  transactions 

1.5  Unique  sequence 
number  generation 

5 

3 

15 

Only  Serial/Serial 8  column 
datatype  -  only  one  per 
table  allowed 

1 .6  Unrestricted  row- 

level  locking  without 
lock  escalation 

10 

6 

60 

43 

37 

267 

2.0  Reliability 

Not  Applicable 

3.0  Maintainability 

3 . 1  Cost  Of  Ownership 

5 

5 

25 

License 

-  Minimum  10  seats 

-  Cost  $12600 

Technical 

Support/Upgrades 

-  Business  hours 

-  Unlimited  calls 

-  Technical  support  cost 
$2695 

-  Update  subscription 
free  with  support 

3.2  Graphical  installer 
and  ease  of 
installation 

5 

6 

30 
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Table  A-2-1  Informix  INFORMIX-Universal  Server  for  Window  NT  v9.11  Beta 

(Continued) 

3 . 3  Documentation 

7 

6 

42 

IPDF  based 

-  Not  capable  of 
searching  multiple 
documents 

-  Good  technical 
information 

-  Some  links  provided 

3.4  Systems  Manager- 
GUI  and  easy  to  use 

6 

6 

36 

V".  ' 

23 

23 

133 

4.0  Producibility 

;  '  : 

Not  Applicable 

5.0  Capability 

‘  <-  >■'  ’  *v„' ;  .  ••  - ' 

5.1  Database  Triggers 

5.1.1  Triggers  execute  on 
INSERT,  UPDATE, 
or  DELETE  either 
BEFORE  or  AFTER 
operations 

7 

8 

56 

Can  also  place  triggers  on 
select  statements 

5.1.2  Triggers  fire  once  per 
statement  or  once  per 
row 

7 

6 

42 

’ 

14 

14 

98 

5.2  Declarative  Integrity 
Constraints 

v 

5.2.1  Cascade  updates  and 
deletes 

7 

3 

21 

Delete  only 

5.2.2  Constraint  checking 
at  end  of  statements 

7 

6 

42 
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Table  A-2-1  Informix  INFORMIX-Universal  Server  for  Window  NT  v9.11  Beta 

(Continued) 

5.2.3  PRIMARY, 

FOREIGN,  and 
UNIQUE  keys, 
CHECK,  DEFAULT, 
and  not  NULL 
constraints 

10 

6 

60 

* 

24 

15 

123 

5.3  Miscellaneous 

5.3.1  CMMS  Dependencies 

10 

3 

30 

Revere  product  only 

5.3.2  CORBA  HOP  support 

6 

0 

0 

None 

5.3.3  LMIS  product 
experience 

10 

0 

0 

None 

5.3.4  View  support 

5 

6 

30 

5.3.5  Extendable  Object- 
Relational  Database 
Architecture 

10 

10 

100 

IUS  is  extendable  by 
adding  DataBlade  modules. 
The  database  server  is 
intrinsically  aware  of  new 
datatypes  added  to  the 
system.  It  has  a  Table- 
driven  extensible  query 
optimizer  and  an  extensible 
indexing  system. 

41 

19 

160 
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Table  A-2^1  Informix  INFORMIX-Universal  Server  for  Window  NT  v9.11  Beta 

(Continued) 

5.4  Object  Types  Support  L 

5.4.1  User-defined  types 
(UDTs)  support  for 
hierarchies  of  types, 
inheritance,  and 

Object  ID  reference 
pointers 

10 

8 

80 

Distinct  types,  opaque 
abstract  datatypes  (ADTs), 
row  types,  and  collection 
types,  including  unlimited 
nested  tables.  IUS  supports 
single  inheritance  among  a 
hierarchy  of  named  row 
types.  In  addition,  any 
table  that  is  based  on  a  row 
type  becomes  a  typed  table 
that  can  also  be  part  of  a 
hierarchy.  Reference  types 
and  SQL3  ADTS  are 
coming,  as  is  replication  of 
UDT  data. 

5.4.2  Internal  and  external 
support  for  large 
objects  (LOBs) 

10 

8 

80 

LOBs  can  be  stored  inside 
the  database  (in  smart 
BLOBs)  or  in  external 
files.  IUS  does  not 
guarantee  the  integrity  of 
data  stored  in  external  files, 
but  Informix  has  future 
plans  in  this  area.  A  virtual 
table  interface  allows 
external  data  to  be 
registered  in  the  database 
catalogs  and  accessed  as  if 
they  were  IUS  tables. 
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Table  A-2-1  Informix  INFORMIX-Universal  Server  for  Window  NT  v9.11  Beta 

(Continued) 

5.4.3  User-defined 

functions  (UDFs) 
with  function 
overloading 

10 

6 

60 

A  full  range  of  UDF  types 
plus  support  for 
overloading,  function 
resolution  based  on 
multiple  attributes,  and 
parallel  execution  of 
functions  where 
appropriate. 

30 

22 

220 

5.5  Programmatic 

Interfaces 

5.5.1  ODBC  and  JDBC 

10 

6 

60 

5.5.2  Support  for  3 GLs, 
4GLs,  and  Object- 
oriented  languages 

10 

6 

60 

20 

12 

120 

5.6  Security,  Roles  and 

Privileges 

5.6.1  Encrypted  passwords 
with  choice  of 
internal  or  external 
user  authentication 

5 

3 

15 

Uses  system  account  to 
establish  database  account 
(external  user 
authentication  only) 

5.6.2  Fine-grained  database 
privileges 

6 

5 

30 

5.6.3  Hierarchical  role- 
based  security  for 
group-level  access 
control 

6 

6 

36 

17 

14 

81 

5.7  Stored  Procedures 
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Table  A-2-1  Informix  INFORMIX-Universal  Serv'er  for  Window  NT  V9.ll  Beta 

(Continued) 

5.7.1  Block  structure  flow 
control  with  robust 
exception  handling, 
remote  procedure 
calls  (RPCs) 
protected  by  a 
transparent  two-phase 
commit,  and  static 
and  dynamic  SQL 
support 

10 

6 

60 

5.7.2  Called  from  3GL  and 
other  stored 
procedures,  database 
triggers  and  SQL 
statements 

10 

6 

60 

5.7.3  Cursor  variables  for 
easy  retrieval  of 

1  multi-row  result  sets 

5 

6 

,  ‘  - .  ■ 

25 

18 

150 

TOTAL  237  174  1352 
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3.2  Oracle  Trade  Study  Assessment  Tables 


Table  A-2-2  Oracle  ORACLE8  Windows  NT  Enterprise  v8.03 

CRITERIA 

RQMTS 

O 

a 

SCORE 

3  O  H 

COMMENTS 

1.0  Performance 

1 . 1  Cost-based,  syntax- 

independent 
optimization 

6 

6 

36 

Statistics  are  kept  current 
by  manually  issuing 
analyzing  schema 
command(s).  Query  plan 
will  be  affected  if  statistics 
are  too  far  out  of  date. 

1 .2  Shared  database 
buffer,  data 
dictionary,  SQL 
statements,  and 
stored  procedures 
cache 

6 

6 

36 

1 .3  B-tree  single  and 
concatenated 
indexes 

6 

6 

36 

1 .4  Contention-free, 
non-blocking, 
multi-version,  and 
read-consistent 
queries 

10 

7 

70 

Transactions  are  implicit 

1 .5  Unique  sequence 
number  generation 

5 

6 

30 

1 .6  Unrestricted  row- 
level  locking 
without  lock 
escalation 

10 

6 

60 

43 

37 

268 
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Table  A-2-2  Oracle  ORACLE8  Windows  NT  Enterprise  v8.03  (Continued) 

•  '•  v'  -  ••  ' '  ‘  T  uu: 

2.0  Reliability 

p.;  ‘ 

"  V  • 

Not  Applicable 

3.0  Maintainability 

«‘?.V  •'  •  • 

3.1  Cost  Of  Ownership 

5 

6 

30 

License 

Minimum  8  seats 

-  Cost  $7960 

Technical 

Support/Upgrades 

-  Normal  business  hours 

-  Unlimited  calls 

-  Technical  support  cost 
$1600 

-  Update  subscription 
free  with  support 

3.2  Graphical  installer 
and  ease  of 
installation 

30 

3.3  Documentation 

7 

5 

35 

HTML  based 

Not  capable  of 
searching  multiple 
documents 

-  Good  technical 
information 

-  Some  broken  links 

3.4  Systems  Manager- 
GUI  and  easy  to  use 

6 

6 

36 

' 

4.0  Producibility 

23 

23 

131 

'  ;  '  ; 

Not  Applicable 
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Table  A-2-2  Oracle  ORACLE8  Windows  NT  Enterprise  v8.03  (Continued) 

5.0  Capability 

5.1  Database  Triggers 

5.1.1  Triggers  execute  on 
INSERT, 

UPDATE,  or 
DELETE  either 
BEFORE  or 

AFTER  operations 

7 

42 

5.1.2  Triggers  fire  once 
per  statement  or 
once  per  row 

7 

6 

42 

14 

12 

84 

5.2  Declarative  Integrity 

Constraints 

5.2.1  Cascade  updates 
and  deletes 

7 

3 

21 

Delete  only 

5.2.2  Constraint  checking 
at  end  of  statements 

7 

7 

49 

Also  can  check  constraint 
at  end  of  transactions 

5.2.3  PRIMARY, 
FOREIGN,  and 
UNIQUE  keys, 
CHECK, 

DEFAULT,  and  not 
NULL  constraints 

10 

6 

60 

24 

16 

130 

5.3  Miscellaneous 

5.3.1  CMMS 

Dependencies 

10 

10 

100 

Revere,  TSW,  and  GOLD 
products 

5.3.2  CORBA  HOP 
support 

6 

6 

36 

5.3.3  LMIS  product 
experience 

10 

10 

100 

5.3.4  View  support 

5 

6 

30 
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5.4  Object  Types  Support 


5.4.1  User-defined  types 
(UDTs)  support  for 
hierarchies  of  types, 
inheritance,  and 
Object  ID  reference 
pointers 


10  I  4 


Oracle8's  extended  type 
system  supports  object, 
collection  (varying  arrays 
and  nested  tables),  and 
reference  types.  An  object 
type  can  apply  to  either  a 
column  or  a  row  and  can 
be  semantically  equivalent 
to  a  SQL3-named  row 
type.  Oracle8  also 
explicitly  associates 
methods  with  object  types. 
Only  one  level  of  nested 
tables  is  supported  in  8.0, 
which  limits  object¬ 
modeling  capability. 
Oracle8  does  not  yet 
support  the  notion  of  a 
fully  encapsulated  abstract 
datatypes.  Oracle8  will 
support  single  inheritance 
with  Database  Extensibility 
Services.  Extended  types 
cannot  be  replicated. 


No  inheritance;  single 
inheritance  coming  in  8.1 
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Table  A-2-2  Oracle  ORAC  LES  Windows  N1 

Enterprise  v8.03  (Continued) 

5.4.2  Internal  and 

external  support  for 
large  objects 
(LOBs) 

10 

4 

40 

Large  objects  can  be  stored 
inside  the  database  or  in 
external  files.  Oracle8  does 
not  support  write  access  to 
or  guarantee  the  integrity 
of  external  data.  LOBs  can 
be  replicated,  but  tables 
with  LOBs  cannot  be 
partitioned. 

5.4.3  User-defined 

functions  (UDFs) 
with  function 
overloading 

_  _  _ 

10 

6 

60 

Scalar  UDFs,  overloading, 
function  resolution  based 
on  multiple  attributes,  and 
parallel  execution  of  UDFs 
with  user-defined 
aggregates  (column 
functions)  is  under 
consideration. 

30 

14 

140 

5.5  Programmatic 

Interfaces 

5.5.1  ODBC  and  JDBC 

6 

60 

5.5.2  Support  for  3GLs, 
4GLs,  and  Object- 
oriented  languages 

10 

6 

60 

20 

12 

120 

5.6  Security,  Roles  and 

Privileges 

5.6.1  Encrypted 

passwords  with 
choice  of  internal  or 
external  user 
authentication 

5 

6 

30 

5.6.2  Fine-grained 

database  privileges 

6 

7 

42 
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Table  A-2-2  Oracle  ORACLE8  Windows  NT  Enterprise  v8.03  (Continued) 

5.6.3  Hierarchical  role- 
based  security  for 
group-level  access 
control 

6 

6 

36 

'  -  *  -  ‘\yh>-  ’•*> 

17 

19 

108 

5.7  Stored  Procedures 

t- 

y&I 

5.7.1  Block  structure 
flow  control  with 
robust  exception 
handling,  remote 
procedure  calls 
(RPCs)  protected  by 
a  transparent  two- 
phase  commit,  and 
static  and  dynamic 
SQL  support 

10 

8 

80 

Ada  like  syntax  and 
structure 

5.7.2  Called  from  3GL 
and  other  stored 
procedures, 
database  triggers 
and  SQL  statements 

10 

6 

60 

5.7.3  Cursor  variables  for 
easy  retrieval  of 
multi-row  result 
sets 

5 

30 

■ :  ■: ; '. :  -fs  •  •  V:  V- ^ *  v ,  * y 

25 

20  | 

170 

total 

237 

190 

1467 
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3.3  Sybase  Trade  Study  Assessment  Tables 


Table  A-2-3  Sybase  Adaptive  Server  Enterprise  Windows  NT  vll.5  Beta 

CRITERIA 

RQMTS 

WEIGH 

SCORE 

TOTAL 

COMMENTS 

1.0  Performance 

1 . 1  Cost-based,  syntax- 
independent 
optimization 

6 

7 

42 

The  server  for  each  query 
analyzes  affected  tables 
automatically.  So  query 
plan  should  not  be  affected 
by  out  of  date  statistics  but 
additional  overhead  created 
by  gathering  for  each  query 
could  possibility  slow  the 
query. 

1 .2  Shared  database 
buffer,  data 
dictionary,  SQL 
statements,  and 
stored  procedures 
cache 

6 

6 

36 

1.3  B-tree  single  and 

concatenated  column 
indexes 

6 

6 

36 

1 .4  Contention-free, 

non-blocking,  multi¬ 
version,  and  read- 
consistent  queries 

1 

0 

6 

60 

By  default,  each  insert, 
update,  and  delete 
statement  is  considered  a 
single  transaction. 

For  contention-free,  non- 
blocking,  multi-version, 
and  read-consistent  queries 
you  have  to  explicitly 
define  transactions 
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3.0  Maintainabili 


3.1  Cost  Of  Ownership 


3.2 

Graphical  installer 

and  ease  of 

installation 

3.3 

Documentation 

3.4 

Systems  Manager- 

GUI  and  easy  to  use 

Table  A-2-3  Sybase  Adaptive  Server  Enterprise  Windows  NT  vll.5  Beta 

(Continued)  ,v~‘  .V  >  .  :i 


1.5  Unique  sequence  5  3  15  Only  IDENTITY  column 

number  generation  datatype  -  only  one  per 

table  allowed _ 


1.6  Unrestricted  row-  10  0  0  Currently  in  alpha  test 

level  locking  without 
lock  escalation 


2.0  Reliabili 


Not  Applicable 


License 

-  Minimum  8  seats 

-  Cost  $3595 
Technical 
Support/Upgrades 

-  Normal  business  hours 

-  Ten  unique  problems 

-  Technical  support  cost 
$1750 

-  Update  subscription 
cost  $645 


A  known  syntax  error 
occurred  but  did  not  affect 
installation 


Dyna  Text  based 
-  Capable  of 

searching  multiple 
documents 
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Table  A-2-3  Sybase  Adaptive  Server  Enterprise  Windows  NT  vll.5  Beta 

(Continued) 

5.0  Capability 

5.1  Database  Triggers 

5.1.1  Triggers  execute  on 
INSERT,  UPDATE, 
or  DELETE  either 
BEFORE  or  AFTER 
operations 

7 

3 

21 

Only  after  the  data 
modification  statement  has 
completed  and  Adaptive 
Server  Enterprise  has 
checked  for  any  datatype, 
rule,  or  integrity  constraint 
violations. 

5. 1 .2  Triggers  fire  once  per 
statement  or  once  per 

row 

7 

3 

21 

Only  once  per  statement 

14 

6 

42 

5.2  Declarative  Integrity 

Constraints 

5.2.1  Cascade  updates  and 
deletes 

7 

0 

0 

None 

5.2.2  Constraint  checking 
at  end  of  statements 

7 

6 

42 

5.2.3  PRIMARY, 

FOREIGN,  and 
UNIQUE  keys, 
CHECK,  DEFAULT, 
and  not  NULL 
constraints 

10 

60 

You  can  also  define  a 
constraint  called  a  rule. 

Once  you  create  a  rule,  you 
can  bind  it  to  multiple  table 
columns  and  to  user 
datatypes. 

24 

102 

5.3  Miscellaneous 

5.3.1  CMMS  Dependencies 

10 

3 

30 

Revere  product  only 

5.3.2  CORBA  HOP  support 

■S 

fMSM 

None 

5.3.3  LMIS  product 
experience 

H| 

H 

U 

None 

5.3.4  View  support 

5 

6 
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Table  A-2-3  Sybase  Ac 

laptive  Server  Enter 
(Continued) 

prise  W 

indows  NT  vll.5  Befj a  •• . 

5.3.5  Extendable  Object- 
Relational  Database 
Architecture 

10 

3 

30 

Separate  server  single 
operational  model 

=  ■  1'  It# Vr  Wi 

41 

12 

90 

5.4  Ob  ject  Types  Support 

||I  ''  ' 

5.4.1  User-defined  types 
(UDTs)  support  for 
hierarchies  of  types, 
inheritance,  and 

Object  ID  reference 
pointers 

■ 

10 

0 

0 

Will  support  Java  UDTs 
first  and  then  SQL3 
datatypes  in  future  versions 

5.4.2  Internal  and  external 
support  for  large 
objects  (LOBs) 

10 

4 

40 

Support  access  to  external 
data  stored  in  separate  data 
stores  through  the 

Component  Integration 

Layer.  Some  partners  store 
data  and/or  indexes  inside 
a  Sybase  database,  and 
others  do  not.  Support  for 
SQL3  LOBs  is  planned 

5.4.3  User-defined 

functions  (UDFs) 
with  function 
overloading 

10 

0 

0 

Future  releases  will  support 
Java  UDFs  returning  scalar 
values  or  Java  object 
references 

' 

30 

4 

40 

5.5  Programmatic 

Interfaces 

\\>  -  -W:.>  ■  • '  5 

"S'. 

Ur,  •  \ 

5.5.1  ODBC  and  JDBC 

10 

6 

60 

5.5.2  Support  for  3 GLs, 
4GLs,  and  Object- 
oriented  languages 

H 

60 

4  ,<v. 

20  | 

12  | 

120 
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Table  A-2-3  Sybase  Adaptive  Server  Enterprise  Windows  NT  vll.5  Beta 

(Continued) 

5.6  Security,  Roles  and 

Privileges 

5.6.1  Encrypted  passwords 
with  choice  of 
internal  or  external 
user  authentication 

5 

4 

20 

Yes  -  unable  to  get  external 
user  authentication  to  work 

5.6.2  Fine-grained  database 
privileges 

6 

4 

24 

No  -  only  create  database, 
default,  procedure,  rule, 
table,  and  view  at  database 
level 

5.6.3  Hierarchical  role- 
based  security  for 
group-level  access 
control 

6 

6 

36 

17 

14 

80 

5.7  Stored  Procedures 

5.7.1  Block  structure  flow 
control  with  robust 
exception  handling, 
remote  procedure 
calls  (RPCs) 
protected  by  a 
transparent  two-phase 
commit,  and  static 
and  dynamic  SQL 
support 

10 

4 

40 

No  exception  handling 

No  dynamic  SQL 

RPC  -  Programmatically 
through  Open  Server 

5.7.2  Called  from  3GL  and 
other  stored 
procedures,  database 
triggers  and  SQL 
statements 

10 

6 

60 

Ill 


Table  A-2-3  Sybase  Ad 

aptive  Server  Enterprise  Windows  NT  vll.5  Beta 
(Continued) 

5.7.3  Cursor  variables  for 
easy  retrieval  of 
multi-row  result  sets 

5 

30 

.■  V :•  :•  >  -v; 

;;v  ■.  x- '  . .■-xx-..  -  ■.  •  -  .  x, 

-.V  ':X  ■:  ■ 

25 

16 

130 

TOTAL  237  127  928 

S 

3.4  Trade  Study  Assessment  Legends 

The  DBMS  trade  study  was  scored  using  the  following  legends: 


Weight  Legend 

Definition 

10 

9 

8 

Feature  required  to  meet  functional  requirements 

7 

Feature  not  needed  to  meet  program  requirements  but  significantly 

6 

5 

reduces  cost,  time,  or  risk 

4 

Feature  not  needed  to  meet  program  requirements  but  reduces  cost, 

3 

2 

1 

time,  or  risk 

Score  Legend 

St  on 

Definition 

10 

Feature  Greatly  Exceeds  Expectations 

9 

8 

Feature  Exceeds  Expectations 

7 

6 

5 

Feature  Meets  Expectations 

4 

3 

2 

1 

0 

Feature  Does  Not  Meet  Expectations 

Feature  Not  Present 
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4.0  Product  Experience 

The  DBMS  packages  were  installed  and  exercised  at  LMIS.  They  were  all  adequate  for  straightforward 
RDBMS  applications.  It  was  unfortunate  that  Informix  did  not  provide  their  DBMS  server  in  time  for 
more  in-depth  hands  on  evaluation  and  provide  enough  technical  support  to  support  the  evaluation. 

5.0  Source  Summary 

The  information  on  which  the  evaluation  was  based  was  drawn  from  a  variety  of  sources.  Material  from 
all  three  vendors  Web  sites,  technical  documentation  provide  by  the  vendors,  and  Web  sites  that  cater 
to  DBMSs  such  as  DBMS  magazine  Web  site  http://www.dbmsmaq.com  provided  most  of  the  technical 
information  contained  in  this  trade  study.  The  vendors’  sales  organizations  and  technical  personnel 
were  also  contacted.  Other  information  was  drawn  from  Lockheed  Martin  Information  Systems 
employees  and  ITI-ALC  Team  members. 

5.1  Product  Data  Sheets 

All  product  data  sheets  collected  for  this  trade  study  will  be  maintained  in  the  Database  Management 
Systems  notebook. 

5.2  Marketing  Information 

All  marketing  information  collected  for  this  trade  study  will  be  maintained  in  the  Database  Management 
Systems  notebook. 

5.3  Points  of  Contact 


Company 

Phone 

URL 

Informix  Software  Inc.,  Menlo 
Park,  CA 

415-926-6300 

http  ://www.informix.com 

Oracle  Corp.,  Redwood  Shores, 

CA 

415-506-7000 

http://www.oracle.com 

Sybase  Inc.,  Emeryville,  CA 

800-685-8225 

http://www.svbase.com 
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A -3  ELECTRONIC  IDENTIFICATION  TRADE  STUDY 
1.0  Summation 

1.1  Purpose 

The  purpose  of  this  trade  study  is  to  evaluate  technologies  and  products  for  the  ITI-ALC  demonstration 
environment  for  electronic  identification  of  individuals.  Technologies  and  products  are  desired  that  are 
easy  to  use  and  acceptable  to  users,  as  well  as  robust  against  misidentifications. 


1.2  Products 

The  methodology  was  to  first  evaluate  the  trade  study  space  consisting  of  available  technologies.  The 
candidate  technologies  were  gathered,  and  leading  contenders  were  selected.  Selected  implementing 
packages  were  brought  in  and  exercised. 

Candidate  technologies  were  picked  to  cover  the  spectrum  from  mature  to  newly  emerging  technologies. 
The  technology  types  are: 

•  Password 

•  Signature 

•  Token  technologies 

-  Proximity  card 

-  Smart  card 

-  Swipe  card 

•  Biometric  measurement  technologies 

-  Voice 

-  Face 

-  Hand  geometry 

-  Fingerprint 

-  Retinal  vascular  pattern 

-  Iris  pattern 

1.3  Environment 

The  electronic  identification  technologies  will  be  used  both  in  an  office  setting  as  well  as  in  a  hangar 
environment.  The  hangar  environment  is  the  more  demanding,  and  its  users  are  those  with  which  the 
ITI-ALC  Phase  2  program  will  most  concern  itself.  The  hangar  environment  can  be  cramped,  noisy, 
have  temperature  extremes,  poorly  lighted,  and  leave  residue  on  users’  hands.  Users  will  commonly 
wear  safety  glasses.  The  technology  must  perform  quickly  enough  that  the  ability  of  users  swiftly  to  do 
their  job  is  not  degraded.  Ideally,  users  should  be  able  to  identify  themselves  more  swiftly  than  the 
current  means  used;  however,  electronic  identification  also  brings  with  it  the  benefits  of  a  paperless 
environment:  easy  access  to  data,  real  time  updating  of  job  completion,  compact  storage  of 
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maintenance  history.  These  benefits  may  outweigh  a  slight  degradation  in  speed  of  use.  The 
environment  includes  a  psychological  component — users  might  have  concerns  about  infringement  on 
their  privacy.  The  use  environment  is  taken  to  be  collegial  rather  than  adversarial.  Users  are  not 
expected  to  mount  malicious  attacks  against  the  recognition  tool.  Indeed,  the  current  system  of  small 
stamps  informally  allows  a  small  amount  of  latitude.  A  user  might  on  occasion  use  someone  else’s 
stamp,  say  to  save  the  need  of  descending  from  an  aircraft  to  stamp  a  card  or,  as  will  certainly 
occasionally  happen,  because  the  stamp  has  been  left  at  home.  This  of  course  is  done  only  when  the 
holder  of  the  stamp  is  familiar  with  the  capabilities  of  the  using  individual,  or  has  looked  over  the  work 
done. 

The  platforms  on  which  these  technologies  will  run  are  PCs.  Potential  configurations  are  wearable, 
hand-held,  slate,  and  desktop  devices.  The  user  database  is  maintained  on  UNIX  servers  or  Windows 
NT  servers.  The  servers  host  the  database  of  individuals  and  perform  the  comparison  of  an  individual’s 
identification  information  with  the  set  of  users.  The  demonstration  environment  assumes  one  server. 
The  full-up  environment  assumes  an  ALC  with  perhaps  1500  users,  three  servers,  and  500  client 
devices. 

An  alternative  architecture  briefly  considered  was  to  put  template  data  describing  one  or  more 
individuals  on  a  portable  platform.  This  presents  significant  database  configuration  management 
difficulties,  and  thus  was  rejected.  It  is  difficult  enough  to  maintain  a  database  at  two  locations;  doing 
so  on  the  large  number  of  platforms  would  lead  to  significant  operability  problems. 


1.4  Summary  of  Best  Candidates 

Candidate  Overview 

Of  the  technologies  considered,  the  best-rated  candidate  is  the  traditional  password.  Passwords 
provide  unambiguous  identification  with  very  little  data  entry  and  transfer,  the  technology  is  very  simple, 
and  the  required  peripherals  are  an  integral  part  of  the  function  of  all  of  the  platforms.  It  is 
recommended  that  password  authentication  be  provided  even  if  another  means  for  electronic 
identification  is  provided.  Passwords  can  provide  a  secondary  means  of  identification,  if  another 
primary  form  is  not  available.  Passwords  have  a  small  drawback  for  portable  computing,  in  that  the 
user  must  either  enter  the  password  with  an  on-screen  keyboard,  which  is  clumsy  and  slow,  or  enter  it 
as  handwriting,  which  requires  writing  the  password  plainly  on  the  screen.  The  latter  choice  risks 
compromising  the  password.  The  virtual  keyboard  is  less  convenient  for  someone  who  is  not  a  typist 
than  an  ordinary  keyboard,  and  many  users  would  not  be  happy  to  use  even  the  usual  full  keyboard. 
Signature  recognition  has  the  advantage  of  being  a  totally  natural  action  on  the  part  of  the  user.  It 
should  meet  with  minimal  resistance  from  users.  It  requires  no  specialized  peripheral,  and  is 
unaffected  by  noise,  lighting,  or  solvents.  It  is  less  unambiguous  than  card  or  password  technology, 
however. 

Of  the  card  identification  methods,  the  proximity  card  has  the  advantage  of  not  needing  to  physically 
contact  the  reader.  This  provides  packaging  advantages.  It  also  makes  it  less  likely  to  fail  than  the 
other  types  of  cards,  since  it  dirt  on  the  card  will  not  affect  it,  and  there  is  no  wear  on  the  reader. 

Voice  recognition  provides  an  extremely  convenient  means  of  identification.  It  is  hands-off  and  needs 
only  a  microphone,  a  peripheral  that  is  fast  becoming  a  standard  item  on  portable  computers.  It  is  a 
very  acceptably  non-intrusive  biometric,  since  what  is  measured  is  something  that  everybody  offers  to 
the  world  from  the  first  “Good  morning”  of  the  day.  Our  experience  with  the  technology  was  very  good. 
The  other  biometrics  presented  some  difficulties.  In  particular,  there  was  an  immediate  and  distinct 
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resistance  on  the  part  of  the  mechanic  users  to  fingerprint  identification,  probably  because  of  its 
association  with  law  enforcement.  Hand,  retina,  and  iris  identification  are  also  intrusive  but  without  the 
law  enforcement  association  of  fingerprint  identification. 

Face  recognition  had  other  difficulties.  The  leading  package,  Facelt,  did  not  reliably  recognize  faces 
unless  the  recognition  threshold  was  dropped  quite  low.  Its  performance  improved  when  it  was  used 
with  a  high  quality  camera.  It  was  not  as  sensitive  to  camera  frame  rate  as  one  would  have  been  led  to 
believe  from  the  accompanying  literature.  Use  of  the  high-quality  ViCam  camera  yielded  distinctly 
better  results  than  were  experience  with  the  DCVC  camera.  The  technology  is  very  exciting,  and  will 
certainly  mature,  but  the  current  state  of  the  art  is  such  that  it  is  not  ready  for  use  in  a  Air  Logistics 
Center  as  a  primary  form  of  electronic  identification.  It  is  a  valuable  tool  when  used  in  conjunction  with 
other  biometrics  or  a  password,  but  that  value  is  depreciated  in  the  non-adversarial  culture  of  an  Air 
Logistics  Center. 

Voice  recognition  also  had  its  difficulties.  This  technology  was  more  reliable  than  the  Facelt  product. 
Various  versions  of  the  package  were  used,  generally  with  very  good  results,  although  one  version  of 
the  product  reliably  generated  false  positive  returns. 

Multiple  Candidates 

After  looking  at  the  various  alternatives  for  electronic  identification,  it  has  become  apparent  that  each 
alternative  has  advantages  and  disadvantages.  Unlike  some  trades,  it  is  not  necessary  or  even 
desirable  to  limit  the  selection  to  a  single  choice.  Different  work  environments  favor  different 
technologies.  For  engineers  with  a  workstation,  a  password  is  a  quick,  familiar  form  of  authentication. 
When  working  in  a  cramped  cockpit,  voice  recognition  offers  the  advantage  of  hands-off  entry  of 
authentication.  Proximity  cards  offer  the  convenience  of  not  having  to  remember  a  password,  while 
providing  the  familiarity  of  a  token  for  authentication. 

Provision  of  more  than  one  type  of  electronic  identification  has  the  advantage  that  one  can  back  up 
others  when  they  are  not  available.  For  example,  if  a  proximity  card  system  is  used,  password 
identification  might  be  used  when  the  card  has  been  forgotten  or  lost.  This  saves  the  mechanic  from 
having  to  get  a  temporary  card.  If  voice  recognition  is  used,  a  password  might  be  used  on  a  day  when 
the  mechanic  has  a  sore  throat.  To  keep  hardware  requirements  to  a  minimum,  if  more  than  one  form 
of  electronic  identification  is  used,  at  least  one  should  run  without  any  additional  hardware  required. 
This  consideration  mandates  either  a  password  system  or  a  signature  recognition  system. 

Multiple  forms  of  identification  can  provide  a  means  to  enhance  security.  In  general  this  is  not  an  issue 
in  the  collegial,  non-adversarial  environment  of  the  Air  Logistics  Centers,  but  it  might  conceivably  be 
used  for  particularly  critical  sign-offs  or  for  access  to  sensitive  data,  such  as  the  audit  trail  for  an 
aircraft’s  maintenance. 

Use  of  more  than  one  form  of  electronic  identification  has  the  disadvantage  of  requiring  more  effort  to 
administer.  The  effort  is,  however,  less  than  double  to  maintain  two  forms  than  to  maintain  one  form. 
This  is  because  only  one  server  and  user  profile  database  would  be  needed.  Enrollment  and  updating 
could  be  done  for  both  forms  at  once.  Moreover,  the  addition  of  the  second  form  has  the  potential  to 
alleviate  administration  costs  of  the  other  form.  For  example  the  availability  of  a  password  or  signature 
system  might  remove  the  need  to  issue  a  temporary  card  when  a  mechanic  forgets  his  card. 

Overall  Recommendation 

The  overall  recommendation  is  to  use  more  than  one  form  of  identification.  The  simplest  method  to 
implement  and  maintain,  and  the  highest  rated,  is  the  traditional  password.  It  should  definitely  be 
provided  since  it  provides  a  simple  backup  for  other  methods.  For  the  benefit  of  users  that  do  not  like 
entering  alphanumeric  data  into  a  computer,  signature  recognition  is  the  recommended  biometric. 
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Communication  Intelligence  Corporation  (CIC)  is  the  standard  vendor  for  this  technology.  Password 
and  signature  recognition  have  the  virtue  that  the  do  not  require  peripherals  that  are  not  standard  items 
on  portable  computers.  For  quick,  unambiguous  identification,  a  proximity  card  method  is  desirable.  If 
cost  and  producibility  constraints  make  it  not  possible  to  include  it  in  the  portable  platforms,  it  could  be 
made  available  at  high-use  locations,  such  as  the  booths  in  the  hangar,  where  the  readers  do  not  have 
to  be  incorporated  into  the  platforms.  Voice  recognition  provides  a  hands-free  method  that  uses  a 
common  peripheral  on  the  coming  generation  of  portable  computers,  a  microphone.  We  had  good 
experience  with  Nordra  Technologies’  Voice  Print  product. 

1.5  Future  Considerations 

The  electronic  identification  market  place  is  dynamic.  It  is  worthwhile  to  keep  track  of  the  state  of  the 
products  available,  since  the  current  leading  products  and  the  condition  and  responsiveness  of  their 
producing  companies  have  the  potential  for  swift  change. 

The  physical  size  of  the  devices  that  read  biometrics  such  as  fingerprint  scanners  or  digital  cameras  is 
getting  smaller,  and  the  price  of  the  devices  is  also  falling.  The  sophistication  of  the  algorithms  is  also 
improving,  although  not  so  rapidly.  As  network  speed  increases,  more  data  can  practically  be  moved 
about  for  identification,  a  consideration  in  the  case  of  face  recognition. 


2.0  Trade  Study  Information 

Methodology 

The  study  was  done  in  two  stages.  The  first  was  to  evaluate  the  candidate  technologies.  The  second 
was  to  evaluate  specific  implementations  of  selected  technologies. 

Various  technologies  and  products  were  evaluated  in  the  context  of  an  Air  Logistics  Center.  The  first 
step  a  broad  search  for  identification  technologies.  The  intent  was  to  identify  a  full  spectrum  of 
technologies,  from  the  mundane  such  as  passwords,  to  the  exotic,  for  example  face  recognition  using 
infrared  cameras.  Supplement  A-3-3  is  a  list  of  companies  making  the  products  considered. 

Electronic  identification  has  been  developed  for  environments  significantly  different  from  that  of  the  Air 
Logistics  Center.  In  particular,  many  of  the  biometric  technologies  are  designed  to  thwart  concerted 
malicious  intrusion  for  applications  such  as  sensitive  installations  or  ATM  machines.  These 
technologies  are  highly  effective,  but  they  are  typically  designed  for  fixed  location  use,  making  them 
heavy  and  reliant  on  external  power.  Some  of  them  are  adaptable  for  portable  use,  however,  notably 
fingerprints,  since  small  inexpensive  fingerprints  sensors  that  can  be  incorporated  into  a  portable 
computer  are  coming  on  the  market.  These  methods  also  have  the  potential  to  be  unacceptable  to 
users,  since  they  intrude  into  the  user’s  privacy.  These  technologies  also  provide  a  level  of  security 
above  that  needed  for  the  Air  Logistics  Center  application.  Indeed,  their  very  ability  to  unambiguously 
identify  an  individual  is  one  reason  that  users  might  find  them  intrusive. 

Some  of  the  technologies  are  well  understood  by  users,  in  particular  card  technologies.  While  the 
underlying  technologies  might  vary,  the  method  of  use  is  familiar  from  the  ubiquity  of  ATM  machines. 
Another  example  of  a  familiar  identification  method  is  passwords.  Familiarity  is,  however,  context 
sensitive.  For  example,  in  the  Air  Logistics  Center  environment  passwords  are  not  universally  familiar, 
since  many  of  the  users  are  not  familiar  with  computers,  particularly  multi-user  computers.  They  are, 
however,  familiar  with  PIN  numbers  from  ATM  use. 

The  identification  methods  were  selected  for  more  careful  analysis  on  the  basis  of  two  criteria.  One 
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was  “Is  the  method  applicable  to  the  Air  Logistics  Center  hangar  activity?”  and  the  other  was  “Is  it  to  the 
boundary  between  common  and  emerging  technologies?”  Common  identification  methods  do  not  need 
much  evaluation;  cutting  edge  technologies  such  as  infrared  face  recognition  were  not  applicable  to  the 
ALC  activity.  The  intent  was  to  identify  ways  of  performing  identification  that  have  the  potential  to  be 
usefully  deployed  as  part  of  the  ALC  work  flow,  with  a  horizon  of  about  two  years.  The  methods 
chosen  to  demonstrate  were  signature  recognition,  voice  recognition,  and  face  recognition. 

SignOn  Verify 

Communications  Intelligence  Corporation  (CIC)  is  a  leader  in  pen-based  technology,  including 
signature  technology  recognition  is.  They  were  not  receptive  to  providing  an  evaluation  copy,  so  the 
technology  was  obtained  from  SignOn  Systems.  Their  SignOn  Verify  package  provides  a  development 
environment,  along  with  a  demonstration.  We  expected  it  to  run  on  our  slate  top  portable  computers. 
After  some  effort  trying  to  get  it  to  run,  we  discovered,  however,  that  the  product  runs  only  on  platforms 
running  Windows  for  Pen  Computing.  The  company  does  not  plan  to  port  their  product  to  Windows  95 
or  later  operating  systems.  Because  of  this,  signature  recognition  should  be  obtained  from  CIC  if,  as  is 
likely,  an  operating  system  other  than  Windows  for  Pen  Computing  is  selected. 

SignOn  Verify  worked  well  in  our  experience.  Signing  using  the  stylus  was  natural.  Enrollment  is 
simple  and  quick.  The  API  is  well  documented,  straightforward,  and  provides  a  good  spectrum  of 
capabilities. 

Voice  Print 

The  voice  recognition  package  chosen  to  work  with  was  Voice  Print  from  Nordra  Technologies.  Nordra 
Technologies  specializes  in  building  multi-biometric  packages.  Nordra  Technologies  was  unusually 
responsive.  The  version  of  Voice  Print  used  includes  a  licensed  version  of  Facelt.  It  uses  the  basic 
Facelt  technology,  but  with  a  different  interface  for  enrollment  of  users. 

Our  experience  with  Voice  Print  was  very  good.  It  worked  even  when  the  speech  was  in  French  or 
Spanish.  Nordra  Technologies  says  that  voice  recognition  works  best  with  an  independent  microphone; 
the  internal  electronics  of  a  handheld  computer  can  affect  the  audio  chip,  injecting  high  frequency 
whine  above  the  human  hearing  range  that  degrades  voice  recognition  accuracy.  Our  experience 
using  built-in  microphones  was,  however,  comparable  to  that  using  an  external  microphone.  The 
enrollment  interface  is  well  built  and  easy  to  use. 

Our  experience  with  Nordra  Technologies  was  also  very  good — they  have  been  extremely  responsive. 
They  have  experience  in  packaging  biometrics  to  enhance  their  ease  of  use,  including  face  recognition 
and  fingerprints.  They  were  very  responsive  in  modifying  Voice  Print  for  ITI-ALC  program 
demonstration  needs. 

Facelt 

Face  recognition  can  be  done  using  visible  or  infrared  radiation.  The  system  using  infrared  radiation 
provides  a  biometric  that  is  harder  to  spoof,  but  it  is  much  more  expensive  and  larger  than  the  system 
using  visible  radiation.  Its  increased  security  is  not  needed  in  the  Air  Logistics  Center  environment. 
The  product  chosen  was  Facelt,  the  current  industry  leader.  Facelt  includes  several  refinements.  The 
product  can  update  the  user  template  slowly  over  time  to  take  into  account  changes  in  appearance,  for 
example  the  growth  of  a  beard.  It  includes  the  ability  to  save  to  an  audit  file  the  pictures  of  individuals 
not  recognized.  The  package  includes  a  well  thought  out  enrollment  sequence.  Separate  thresholds 
are  user-configurable  for  extracting  faces  and  for  recognizing  faces.  Facelt  can  make  use  of  small 
head  movements  to  enhance  its  ability  to  extract  a  face  from  background  clutter.  Use  of  this  feature  is 
user  selectable.  It  also  can  sense  small  facial  movements,  particularly  eye  and  mouth  movement,  to 
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ensure  that  the  face  presented  is  not  a  simulacrum. 

Our  experience  with  Facelt  was  mixed.  The  initial  camera  used  was  the  Digital  Vision  DCVC2  camera 
recommended  by  Facelt.  The  recommendation  was  to  use  a  PCI  video  board,  but  that  is  not  feasible 
with  a  portable  computer.  The  alternative  taken  was  to  use  a  PCMCIA  card  from  the  camera  vendor, 
which  is  less  desirable,  since  the  interface  provided  is  not  as  fast.  We  also  experienced  overheating  of 
the  card  and  consequent  unreliable  performance.  With  this  configuration,  Facelt  works  fairly  well,  but  it 
was  not  quick  at  extracting  faces,  and  needed  a  fairly  low  threshold  before  it  would  recognize  faces 
reliably.  We  experienced  better  performance  using  a  Vista  Imaging  ViCam  camera.  This  camera  is 
superior  in  performance  to  the  other,  but  slower,  since  it  uses  a  parallel  port.  Another  advantage  is  that 
it  is  powered  by  the  port  and  does  not  require  an  external  power  supply.  Our  experience  with  Facelt 
was  better  using  this  camera.  Its  superior  video  quality  was  more  than  enough  to  overcome  the 
disadvantage  of  a  lower  frame  rate.  Facelt  performance  is  thus  sensitive  to  the  camera  characteristics. 
We  found  that  face  extraction  was  difficult  in  lighting  that  varied  sharply  in  intensity.  In  particular,  there 
were  situations  in  which  a  strong  light  behind  the  face  made  face  extraction  less  effective. 

A  drawback  in  our  experience  was  that  Facelt  was  slow  compared  to  other  biometric  technologies.  We 
also  found  that  its  performance  is  very  sensitive  to  the  face  extraction  and  recognition  thresholds.  This 
implies  that  management  might  be  difficult — if  the  thresholds  are  set  low  enough  that  recognition  occurs 
quickly  and  reliably,  there  seemed  to  be  a  significant  number  of  false  positive  identifications.  For  some 
applications,  this  is  not  necessarily  a  drawback.  One  way  of  using  biometrics  is  to  use  several,  so  that 
if  one  fails,  another  can  be  used.  For  high  assurance  of  identification,  all  methods  must  recognize  the 
individual;  for  lower  levels  some  combination  of  methods  can  be  deemed  to  be  sufficient. 

The  overall  impression  is  that  face  recognition  is  an  exciting  technology  that  does  indeed  work,  but  that 
it  was  relatively  slow  and  unreliable  in  identification.  Face  recognition  technology  is  advancing  rapidly, 
as  is  the  speed  of  computation,  however,  so  that  a  revisit  of  face  recognition  methods  in  a  year  or  two 
might  find  a  package  that  is  quick  and  reliable. 
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3.0  Trade  Study  Assessment  Table 


3.1  Evaluation  Criteria 

For  each  of  the  technologies  and  packages,  a  Kempner-Tregoe  trade  study  was  executed.  This  entails 
developing  a  set  of  evaluation  criteria.  The  ITI-ALC  team  reviewed  the  criteria,  and  any  comments 
were  addressed  by  modifications  to  the  criteria.  The  criteria  were  grouped,  and  each  group  of  criteria 
was  assigned  a  weight.  Each  criterion  in  each  group  was  assigned  an  intra-group  weight.  From  these, 
an  overall  weight  was  developed  for  each  criterion,  as  described  in  Supplement  A-3-2.  This  method  of 
developing  criterion  weights  was  used  so  that  the  criteria  groups  would  be  weighted  as  desired.  It 
avoids  inadvertently  giving  a  criterion  group  too  much  weight  because  there  are  many  criteria  in  it,  or 
giving  a  criterion  group  too  little  weight  because  there  are  few  criteria  in  it.  The  resulting  weights  are 
those  that  appear  in  the  criterion  weights  in  Supplement  A-3-1  and  in  Table  A-3-1. 

Evaluation  criteria  and  their  weights  were  created  and  evaluated  from  two  perspectives.  One  was  the 
applicability  of  the  candidate  technology  or  product  to  fulfill  the  demonstration  needs  of  the  ITI-ALC 
program  over  its  three-year  life.  The  other  was  the  applicability  to  a  production  version  of  the 
capabilities  demonstrated  during  the  ITI-ALC  program.  The  latter  view  was  driven  by  the  desire  to 
make  whatever  is  developed  in  the  ITI-ALC  program  as  reusable  as  possible.  This  has  implications  in 
evaluating  the  corporate  strength  of  vendors,  the  scalability  of  the  technology  and  products,  and  cost 
structure. 

The  evaluation  criteria  are  described  in  Supplement  A-3-1,  with  the  weights  assigned  to  each  criterion. 
Security  was  removed  as  a  criterion,  since  the  Air  Logistics  Center  environment  is  collegial,  and  a 
concerted  attack  on  the  security  of  the  system  was  considered  to  be  unlikely.  Producibility  is  subsumed 
in  the  criteria  addressing  cost— all  of  the  technologies  are  technically  producible. 

3.2  Trade  Study  Assessment  Table 

Table  A-3-1  is  the  Trade  Study  Assessment  Table  for  the  products  considered.  This  table  presents  the 
evaluation  criteria,  which  are  described  in  more  detail  in  Supplement  A-3-1 .  It  gives  the  weight  for  each 
criterion,  and  a  raw  score  for  each  product  and  criterion.  From  these,  a  weighted  score  for  each 
product  and  criterion  is  derived,  the  product  of  the  criterion  weight  and  the  raw  product  score.  The 
weighted  scores  for  each  product  are  summed  to  provide  the  overall  measure  for  the  product.  The 
rationale  for  the  assignment  of  raw  scores  for  each  criterion  is  given  below. 
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Table  A-3-1  Electronic  Identification  Technology  Trade  Study  Assessment 


Raw  Product  Scores  Weighted  Product  Scores 


<Cards  > 

<— 

-  Biometrics  — 

> 

!<-- 

Cards 

— > 

j< — 

Biometrics 

— > 

% 

Password 

Signature 

Proximity 

Smart 

Swipe 

Voice 

Face 

Hand 

Finger 

Retina 

.C/3 

Password 

Signature 

Proximity 

Smart 

Swipe 

Voice 

Face 

Hand 

Finger 

Retina 

’C 

Capabilities 

Time  to  identify 

3 

10 

9 

10 

10 

10 

6 

5 

8 

8 

7 

7 

30 

27 

30 

30 

30 

18 

15 

24 

24 

21 

21 

Weight  of  medium 

2 

10 

10 

9 

9 

9 

10 

10 

10 

10 

10 

10 

20 

20 

18 

18 

18 

20 

20 

20 

20 

20 

20 

Weight  of  reader 

2 

10 

10 

5 

5 

5 

10 

6 

2 

5 

2 

2 

20 

20 

10 

10 

10 

20 

12 

4 

10 

4 

4 

Size  of  medium 

2 

10 

10 

8 

8 

8 

10 

10 

10 

10 

10 

10 

20 

20 

16 

16 

16 

20 

20 

20 

20 

20 

20 

Size  of  reader 

2 

10 

10 

5 

5 

5 

10 

6 

2 

5 

2 

2 

20 

20 

10 

10 

10 

20 

12 

4 

10 

4 

4 

Reader  power  use 

2 

10 

10 

8 

8 

8 

10 

8 

8 

8 

8 

8 

20 

20 

16 

16 

16 

20 

16 

16 

16 

16 

16 

Robust  in  hangar 

2 

10 

10 

10 

7 

7 

8 

6 

10 

10 

6 

6 

20 

20 

20 

14 

14 

16 

12 

20 

20 

12 

12 

Ease  of  Use 

Learning  curve 

2 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

20 

20 

20 

20 

20 

20 

20 

20 

20 

20 

20 

Convenience 

6 

7 

10 

10 

7 

7 

10 

7 

1 

10 

1 

1 

42 

60 

60 

42 

42 

60 

42 

6 

60 

6 

6 

Intrusiveness 

10 

10 

8 

10 

10 

10 

9 

9 

7 

5 

5 

5 

100 

80 

100 

100 

100 

90 

90 

70 

50 

50 

50 

Performance 

Security 

0 

P(misidentification) 

4 

10 

9 

10 

10 

10 

8 

7 

10 

10 

10 

10 

40 

36 

40 

40 

40 

32 

28 

40 

40 

40 

40 

P(nonidentification) 

4 

10 

9 

10 

10 

10 

8 

7 

10 

10 

10 

10 

40 

36 

40 

40 

40 

32 

28 

40 

40 

40 

40 

Data  volume  for  ID 

2 

10 

8 

10 

10 

10 

6 

4 

7 

7 

6 

6 

20 

16 

20 

20 

20 

12 

8 

14 

14 

12 

12 

Reliability 

Technology  maturity 

4 

10 

8 

10 

10 

10 

8 

7 

9 

9 

9 

9 

40 

32 

40 

40 

40 

32 

28 

36 

36 

36 

36 

Reliability 

3 

10 

10 

10 

9 

9 

10 

9 

9 

9 

9 

9 

30 

30 

30 

27 

27 

30 

27 

27 

27 

27 

27 

Installed  base 

3 

10 

7 

10 

9 

10 

7 

5 

6 

6 

5 

5 

30 

21 

30 

27 

30 

21 

15 

18 

18 

15 

15 

Vendor  stability 

4 

10 

9 

9 

9 

10 

8 

8 

8 

10 

8 

8 

40 

36 

36 

36 

40 

32 

32 

32 

40 

32 

32 

Cost 

Acquisition  cost 

3 

10 

9 

7 

7 

7 

9 

7 

7 

7 

6 

6 

30 

27 

21 

21 

21 

27 

21 

21 

21 

18 

18 

Maintenance  cost 

4 

10 

9 

8 

8 

8 

9 

9 

9 

9 

9 

9 

40 

36 

32 

32 

32 

36 

36 

36 

36 

36 

36 

Administration 

Ease  of  installation 

3 

10 

10 

8 

8 

8 

10 

8 

2 

8 

8 

8 

30 

30 

24 

24 

24 

30 

24 

6 

24 

24 

24 

Ease  of  management 

4 

10 

10 

8 

8 

8 

9 

8 

9 

9 

9 

9 

40 

40 

32 

32 

32 

36 

32 

36 

36 

36 

36 
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Table  A-3-1  Electronic  Identification  Technology  Trade  Study  Assessment  (Continued) 


TOTAL  WEIGHTED 
SCORES 


692  647 

645  615  622 

624  538  510  510  489  489 

Password 

Signature 

<—  Cards  — > 

•t  t=  - 

’><  £  £ 

O  CO  oo 

PL, 

< - Biometrics - > 

u  "O  c3  2 

.S2  o  c  bQ  .£  .22 

O  cd  o3  c3  ir!  *-« 

>  a  g  £  " 

Capabilities 


Time  to  identify 

This  criterion  is  the  time  it  takes  to  complete  identification  from  the  end  of  user  action.  It  does  not 
include  the  time  it  takes  to  enter  a  password,  swipe  a  card,  or  position  part  or  all  of  the  individual’s  body 
for  biometric  reading.  The  quickest  technologies  are  those  that  need  to  gather  the  least  amount  of 
information  from  the  individual  to  be  identified, 

Two  things  make  for  quick  identification.  The  first  is  to  gather  little  information  from  the  ambient 
environment.  The  second  is  to  require  little  processing  to  determine  if  the  gathered  information 
matches  an  individual’s  profile.  The  password,  proximity  card,  smart  card,  and  swipe  card  technologies 
all  read  only  a  few  bytes  of  information.  In  the  case  of  the  card  technologies,  the  reader  reads  the  data 
information  and  also  redundant  check  information,  so  that  it  may  be  assumed  that  the  information  was 
gathered  without  any  error.  That  is  the  case,  of  course,  for  password  entry,  too.  The  match  of  all  these 
must  be  exact  with  the  user  profile  stored  in  the  identification  database;  the  database  may  be  entered 
on  a  key,  and  thus  identification  can  be  done  extremely  swiftly. 

The  other  technologies  require  gathering  considerably  more  information,  with  attendant  delays  in 
transmitting  it.  Doing  preprocessing  at  the  gathering  platform  can  minimize  the  amount  of  data  to  be 
transmitted,  but  it  will  still  be  larger  than  is  the  case  for  card  or  password  technologies.  In  any  case,  the 
data  gathered  directly  from  the  sensor  must  be  processed  to  extract  characteristic  statistics.  These 
must  then  be  compared  with  the  database  of  statistics  corresponding  to  candidate  personnel.  The 
comparison  is  not  immediate  nor  precise  in  general,  so  the  comparison  returns  a  best  fit.  This  requires 
more  data  access  and  processing  to  find  approximate  fits  and  then  the  best  fit  than  for  card  or 
password  technologies. 

The  time  to  identify  is  longest  for  face  recognition.  This  is  because  the  processing  requires  two  steps, 
both  of  which  are  complicated.  The  first  step  is  extraction  of  the  face  from  the  background;  the  second 
is  comparison  of  the  face  to  the  database.  Of  the  two,  the  former  generally  takes  the  longest.  The 
rapidity  of  face  extraction  depended,  in  our  experience,  primarily  on  the  granularity  of  the  image.  Face 
recognition  was  correspondingly  downrated.  The  other  biometrics  requiring  feature  extraction  from  a 
background,  iris  and  retina,  were  downrated,  although  the  features  to  be  extracted  are  more  regular, 
hence  easier  to  identify.  Hand  and  finger  identification,  on  the  other  hand,  do  not  require  feature 
extraction  from  the  background.  Voice  recognition  requires  a  few  seconds  of  speech. 
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Weight  of  medium 

No  medium  is  required  for  any  of  the  technologies  considered  except  for  the  card  technologies.  The 
weight  of  the  cards  is  minimal,  well  under  an  ounce. 

Weight  of  reader 

This  criterion  addresses  the  weight  of  the  device  gathering  information  from  the  individual.  The  weight 
in  question  is  that  which  is  above  and  beyond  the  weight  of  the  computing  platform  to  which  it  is 
attached,  so  integrated  devices  that  are  standard  on  commonly  available  platforms  have  no  additional 
weight. 

A  password  requires  no  reader  above  and  beyond  the  computing  platform. 

Signature  recognition  systems  can  be  classified  physically  into  two  types.  One  uses  a  specialized  pen 
that  senses  pressure  and  acceleration  or  a  pressure  sensitive  pad.  The  other  uses  the  standard  pen 
that  comes  with  a  pen-based  computing  platform;  this  type  gathers  fewer  data,  but  it  also  requires  no 
hardware  above  and  beyond  that  available  on  the  basic  platform.  This  type  can  also  be  used  on 
mouse-based  machines,  say  those  used  by  engineers.  We  consider  the  second  type  the  logical  choice 
for  ITI-ALC  because  it  is  highly  desirable  to  minimize  the  amount  of  equipment  a  mechanic  must  carry 
about.  Since  the  environment  is  collegial,  the  enhanced  security  provided  by  a  larger  set  of 
measurands  is  not  needed. 

Card  readers  are  small  and  can  be  integrated;  they  weigh  a  few  ounces.  Voiceprints  require  only  a 
microphone,  which  can  weigh  an  ounce  or  less  and  are  now  typically  integrated  into  the  computer,  so 
that  no  additional  weight  can  be  ascribed  to  the  microphone.  Face,  iris,  or  retina  recognition  requires  a 
small  video  camera;  such  cameras  are  available  mounted  on  a  small  card.  Since  cameras  integrated 
into  the  platform  as  a  standard  feature  are  rare  and  likely  to  remain  so,  their  weight,  on  the  order  of  one 
or  two  ounces,  must  be  included.  Iris  and  retina  identification  systems  are  now  currently  oriented  more 
toward  ATM  environments,  where  weight  is  not  an  issue,  and  thus  are  heavier.  Fingerprint  recognition, 
and  card  technologies  require  light  small  readers  that  can  be  integrated  into  a  platform,  while  hand 
geometry  requires  a  reader  weighing  several  ounces.  Commercial  systems  are  larger. 

Face,  iris,  or  retina  recognition  requires  a  small  video  camera;  such  cameras  are  available  mounted  on 
a  small  card.  Since  cameras  integrated  into  the  platform  as  a  standard  feature  are  rare  and  likely  to 
remain  so,  their  weight,  on  the  order  of  one  or  two  ounces,  must  be  included.  It  is  assumed  that  a 
commercial  packaging  will  be  used  rather  than  a  custom  packaging,  so  the  weight  of  the  camera  must 
include  a  housing.  A  fingerprint  reader  is  light,  weighing  on  the  order  of  an  ounce,  including  an  external 
commercial  housing.  The  pad  for  hand  geometry  recognition  requires  a  device  the  size  of  a  hand, 
weighing  several  ounces. 

Size  of  medium 

Biometrics,  passwords,  signature  require  no  medium.  All  of  the  card  technologies  use  a  medium  on  the 
order  of  the  size  of  a  credit  card.  Such  small  media  are  not  inconvenient,  so  the  technologies  using 
them  are  not  correspondingly  downrated. 

Size  of  reader 

This  criterion  addresses  the  size  of  the  device  gathering  information  from  the  user.  The  weight  in 
question  is  that  which  is  above  and  beyond  the  weight  of  the  computing  platform  to  which  it  is  attached, 
so  integrated  devices  that  are  standard  on  commonly  available  platforms  have  no  additional  weight. 
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this  type  gathers  fewer  data,  but  it  also  requires  no  hardware  above  and  beyond  that  available  on  the 
basic  platform.  We  consider  the  second  type  the  logical  choice  for  ITI-ALC  because  it  is  highly 
desirable  to  minimize  the  amount  of  equipment  a  mechanic  must  carry  about.  Moreover,  the 
environment  is  collegial,  rather  than  adversarial,  so  the  reduced  security  provided  by  a  larger  set  of 
measurands  is  not  needed. 

As  described  in  the  Weight  of  reader  section,  the  signature  method  used  is  that  using  the  computer’s 
stylus  or  mouse.  These  are  integral  parts  of  the  commercial  computing  platform,  and  thus  have  no 
additional  size  above  the  computing  platform. 

A  small  microphone  is  normally  integrated  into  the  new  generation  of  portable  computing  platform,  so 
its  size  is  not  an  issue.  Card  readers  are  small  and  can  be  integrated;  their  size  is  normally  on  the 
order  of  one  to  four  square  inches.  Face,  iris,  or  retina  recognition  requires  a  small  video  camera;  such 
cameras  are  available  mounted  on  a  small  card.  Some  commercially  available  platforms  are  available 
which  integrate  a  camera,  but  this  is  currently  uncommon.  A  fingerprint  reader  can  be  a  small  device, 
smaller  than  a  square  inch.  The  pad  for  hand  geometry  recognition  requires  a  device  the  size  of  a 
hand,  and  is  consequently  considerably  larger. 

Reader  power  use 

It  is  assumed  that  the  platform  is  turned  on  and  the  device  and  its  screen  are  active,  but  that 
peripherals,  even  if  they  are  integrated,  are  not  active.  This  is  because  the  manufacturers  of  platforms 
are  responding  to  the  limitations  imposed  by  limited  battery  power  by  intelligently  managing  the  activity 
of  peripherals.  The  assumption  is  that  integrated  peripherals  will  be  so  managed  as  part  of  the  normal 
system  operation,  but  that  peripherals  not  available  as  standard  features  of  the  majority  of  commercial 
platforms  will  require  some  other  form  of  power  management.  There  are  two  sources  of  power 
consumption.  One  is  gathering  the  measurands;  the  other  is  transmitting  them  to  the  server  that 
determines  the  identity  of  the  user. 

Use  of  password  requires  no  additional  power,  and  the  data  that  must  be  transferred  is  very  small,  so 
power  consumption  is  negligible.  Since  we  assume  that  signatures  are  taken  using  the  standard  stylus, 
no  additional  power  is  required  to  gather  a  signature;  the  amount  of  data  captured  in  a  signature  is  on 
the  order  of  at  most  a  few  hundred  bytes,  so  power  consumption  is  negligible. 

Card  readers  use  a  small  amount  of  power  when  they  are  reading  a  card.  However,  they  are  not 
integrated  devices  in  commercially  available  devices.  They  will  in  general  require  power  even  when 
they  are  not  reading,  or  else  the  system  will  require  active  power  management,  turning  them  on  only 
when  it  is  anticipated  that  they  will  be  used.  While  this  is  technically  feasible,  it  will  add  to  the  cost  and 
complexity  of  the  platforms,  and  diminish  their  reliability.  An  alternative  is  to  require  a  user  operation 
when  reading  the  medium  to  turn  the  reader  on  and  off.  This  is  technically  feasible,  but  it  is 
inconvenient.  As  a  result  of  these  considerations,  card  readers  are  downgraded.  The  situation  is 
comparable  for  cameras  needed  for  feature  recognition  in  images,  and  for  fingerprint  and  hand 
configuration  recognition. 

Robustness  in  hangar 

The  hangar  environment  is  the  primary  environment  considered  for  this  evaluation  since  it  presents  the 
most  difficult  of  the  environments  that  will  be  encountered  and  most  of  the  users  work  there. 
Environmental  challenges  include  the  possibility  that  units  will  suffer  shocks  and  blows  during  their  use, 
lighting  extremes  that  range  from  a  bare  bulb  in  the  background  to  darkness  in  enclosed  areas  of  an 
aircraft,  ambient  noise,  temperature  extremes,  temperature  variations,  grease,  dirt,  and  corrosive 
solvents. 


124 


Passwords  must  be  entered  via  a  keyboard  or  by  handwriting.  Keyboards  are  vulnerable  to  liquid  spills 
and  physical  shocks.  Use  of  a  stylus  requires  an  environment  clean  enough  that  the  screen  can  be 
read  and  written.  A  signature  requires  a  comparably  clean  environment.  Of  the  card  technologies,  the 
one  that  is  most  resistant  to  the  environment  is  the  proximity  card  technology.  The  other  card 
technologies  are  sensitive  to  dirt,  particularly  after  the  readers  have  been  used  for  several  years  and 
the  sensing  elements  have  been  subjected  to  wear. 

Devices  that  require  extraction  of  features  from  an  image  are  sensitive  to  lighting.  They  are  downrated, 
since  the  lighting  in  the  hangar  is  highly  variable,  but  also  it  may  exhibit  a  large  amount  of  variation  in  a 
single  image  since  lighting  inside  an  aircraft  is  likely  to  be  poorly  diffused.  We  have  encountered 
difficulty  in  an  office  environment  when  extracting  faces  in  lighting  with  large  sharp  variations  in 
intensity.  We  expect  that  such  lighting  will  present  comparable  difficulty  in  the  hangar  environment. 

The  hangar  environment  is  often  noisy,  with  noise  from  activity  involving  the  actions  of  the  personnel 
working  on  the  aircraft  as  well  as  from  equipment  like  fans,  compressors,  and  other  continuously 
running  equipment.  Voice  recognition  is  correspondingly  downrated.  Voice  recognition  works  best  with 
an  independent  microphone;  the  internal  electronics  of  a  handheld  computer  can  affect  the  audio  chip, 
injecting  high  frequency  whine  that  degrades  voice  recognition  accuracy. 

Ease  of  Use 


Learning  curve 

This  criterion  addresses  the  ease  of  using  the  electronic  identification  methods.  There  are  two  types  of 
learning  required  to  effectively  deploy  an  electronic  identification  system.  One  is  the  learning  that  is 
required  for  administration  of  the  system;  the  other  is  the  learning  that  is  required  from  the  users  of  the 
deployed  system,  the  mechanics,  planners,  and  engineers. 

Users  need  to  learn  how  to  use  electronic  identification  on  a  day-to-day  business.  They  also  need  to 
know  how  to  manage  their  own  identification,  say  how  to  change  a  password,  to  update  their  signature, 
or  to  update  their  voice.  In  general,  they  may  not  need  to  remember  how  to  do  these  rarely  done 
activities,  but  they  do  need  to  know  at  least  where  to  find  instruction  describing  how  to  do  them. 

All  of  the  forms  of  electronic  identification  require  some  type  of  administration  of  the  database  that 
maintains  the  description  of  the  authorized  users  and  their  identities.  To  allow  an  apples-to-apples 
comparison,  it  is  assumed  that  the  data  are  maintained  in  a  database  whose  type  is  the  same  for  all 
technologies,  say  under  a  DBMS,  or  as  a  collection  of  flat  files.  A  multimedia  database  is  somewhat 
harder  to  maintain  than  a  traditional  text-oriented  database,  but  the  difference  is  small  compared  to 
other  learning  curve  difficulties.  The  learning  curve  for  maintaining  the  database  as  a  database  is  thus 
assumed  to  be  the  same  for  all  forms  of  electronic  identification. 

Passwords  require  no  effort  for  the  user  to  learn  to  use  if  data  are  entered  with  a  keyboard.  Even  users 
with  no  typing  skills  can  quickly  learn  where  the  few  letters  of  their  password  are  located.  If  the  data 
are  entered  using  handwriting,  a  certain  amount  of  practice  will  be  required  to  learn  to  write  clearly 
enough  that  the  handwriting  will  be  reliably  recognized.  There  must  be  a  means  of  changing  a  user’s 
password;  indeed  good  practice  requires  a  regular  change.  A  simple  screen  for  changing  a  password 
takes  care  of  this  requirement  and  requires  minimal  learning  by  the  user.  Otherwise  administration  of 
passwords  is  straightforward. 

Entering  a  signature  in  a  box  with  a  stylus  is  a  natural,  simple  action.  If  electronic  identification  is  used 
at  a  platform  without  a  stylus,  two  approaches  can  be  taken.  One  is  to  provide  a  signature  pad;  the 
other  is  to  use  the  mouse.  The  second  approach  will  require  a  few  minutes  practice  learning  how  to 
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sign  with  a  mouse — it  is  not  the  natural  movement  of  a  signature  using  a  stylus.  Administration  of  a 
signature-based  system  is  simple,  although  users  need  to  have  means  to  change  their  signatures,  for 
example  if  someone  takes  on  a  new  surname  at  marriage.  This  will  occur  infrequently,  and  can  be 
ignored  as  an  operational  issue.  Signature  technology,  like  other  biometrics,  requires  setting  the 
threshold  for  acceptance  of  a  signature  as  genuine.  Users  with  high  variability  in  their  signature  might 
require  adjustment  of  the  threshold.  The  package  explored  for  signature  recognition  provides  the  ability 
to  set  different  thresholds  for  different  users.  The  face  and  voice  recognition  packages,  however, 
provided  only  a  single  threshold  for  all  users. 

All  of  the  card  technologies  require  minimal  training  for  users.  Virtually  everyone  in  the  ITI-ALC  target 
community  will  be  familiar  with  ATM  technology.  Administration  of  these  forms  of  identification  requires 
building  and  issuing  cards.  It  also  requires  a  provision  for  allowing  workers  to  sign  off  when  their  card 
has  been  forgotten  or  lost.  Because  of  this,  more  training  will  be  requires  for  the  administrators. 

Physical  biometrics  requires  minimal  training  for  users.  Face,  hand,  finger,  retina,  and  iris  biometrics 
require  only  that  the  user  present  himself  before  the  sensor.  Voice  requires  only  speech,  a  completely 
natural  action. 

Administration  of  the  database  for  face  and  voice  recognition  requires  tuning  the  recognition  threshold. 
Alternative  forms  of  identification  must  be  provided  when  someone  has  a  problem,  say  a  bad  cold, 
which  temporarily  alters  the  voice.  The  same  is  true  for  facial  recognition  if  an  individual  has  something 
on  his  face,  say  a  bandage  that  makes  face  recognition  difficult.  These  contingencies  raise  the  training 
needs  for  the  administrator. 

Convenience 

There  is  convenience  for  the  user  and  convenience  for  the  administrator.  For  this  criterion, 
convenience  for  the  user  is  by  far  the  more  important  since  there  are  so  many  more  users  than 
administrators.  The  only  time  when  administrator  convenience  becomes  an  issue  is  when 
inconvenience  in  administration  spills  over  into  inconvenience  for  the  user. 

Use  of  a  password  is  not  very  convenient  on  platforms  that  lack  a  keyboard.  Keyboard-based 
password  systems  customarily  present  the  typed-in  password  as  a  string  of  asterisks  to  avoid 
compromising  the  password.  This  is  difficult  to  do  on  a  pen-based  system.  Either  the  user  must  write 
the  word  in  clear  for  handwriting  recognition  software  to  translate,  or  the  user  must  use  an  on-screen 
keyboard,  which  is  clumsy  and  slow.  On  keyboard-based  keyboards,  a  password  is  very  quick  if  the 
user  can  type,  and  quick  even  if  typing  skills  are  minimal,  since  the  password  will  be  entered 
repeatedly,  and  the  user  will  quickly  leam  the  location  of  the  letters  in  the  password.  If  passwords  are 
used,  the  ITI-ALC  environment  is  such  that  is  highly  desirable  to  allow  users  to  choose  their  own 
password — the  additional  security  provided  by  automated  password  selection  is  not  needed  in  the 
collegial  ITI-ALC  environment,  and  it  generates  user  resistance. 

A  signature  is  extremely  convenient  for  the  user  of  a  pen-based  platform — it  is  a  natural  motion.  It  is 
less  so  for  a  mouse-based  platform,  although  it  is  usable. 

Cards  are  extremely  convenient  to  use  when  the  user  has  them.  The  drawback  in  convenience  is  the 
necessity  for  having  the  card  present.  The  cards  are  small  enough  that  they  can  easily  be  carried  in  a 
pocket,  and  so  are  unlikely  to  be  left  on  the  hangar  floor.  Their  main  inconvenience  is  that  they  can  be 
forgotten  or  mislaid.  This  requires  a  user  to  look  about  the  work  area  for  the  card,  to  have  to  go 
somewhere  to  get  a  temporary  card  or  to  use  a  secondary  means  of  identification. 

Voice  recognition  is  very  convenient.  Voice  input  is  possible  in  cramped  quarters  or  the  dark.  It 
requires  only  about  10  seconds  of  speech,  which  can  easily  be  repeated.  A  real  inconvenience  is  that 
caused  by  variability  in  voice,  say  from  laryngitis  or  stress.  This  requires  a  secondary  means  of 
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Voice  recognition  is  very  convenient.  Voice  input  is  possible  in  cramped  quarters  or  the  dark.  It 
requires  only  about  10  seconds  of  speech,  which  can  easily  be  repeated.  A  real  inconvenience  is  that 
caused  by  variability  in  voice,  say  from  laryngitis  or  stress.  This  requires  a  secondary  means  of 
identification. 

Facial  recognition  has  significant  inconvenience.  It  requires  adequate  lighting,  and  the  user  must  face 
the  camera  or  hold  the  camera  up  in  front  of  the  user.  Our  experience  with  the  technology  has  been 
that  the  time  it  takes  to  extract  a  face  from  the  background  and  then  to  compare  it  with  the  database  is 
considerable,  on  the  order  of  a  good  minute  or  longer.  In  addition,  the  reliability  of  identification  is  not 
as  high  as  other  technologies,  which  means  that  more  than  one  attempt  may  be  required  to  sign  off.  A 
sharp  change  in  appearance,  such  as  shaving  off  a  beard,  requires  building  a  new  template  for  the 
user.  This  is  inconvenient  as  well  as  requiring  user  training. 

Hand  shape  recognition  is  convenient,  but  the  size  of  the  readers  makes  them  inconvenient  for  use  in 
the  aircraft.  This  is  also  the  case  for  the  current  implementations  of  retina  and  iris  identification.  These 
two  technologies,  like  face  recognition,  require  lighting  and  require  the  user  to  face  the  camera. 
Fingerprint  sensors  have  become  inexpensive  and  small.  They  are  a  convenient  and  unambiguous 
means  of  identification.  Their  main  drawback  is  sensitivity  to  dirty  hands. 

Intrusiveness 
A  password  is  non-intrusive. 

A  signature  is  somewhat  intrusive,  since  a  signature  is  used  for  purposes  other  than  the  work 
environment.  It  presents  the  risk,  albeit  small,  of  use  for  forgery  in  other  environments. 

Card  technologies  are  non-intrusive. 

Biometrics  are  all  to  some  degree  intrusive.  The  perception  of  intrusiveness,  however,  varies.  For 
example,  people  in  the  United  States  are  comfortable  with  having  their  picture  taken,  because  it  is  done 
often  and  in  many  settings,  and  because  a  photograph  is  not  an  unambiguous  means  of  identification, 
and  because  a  person’s  face  is  a  natural  form  of  identification  in  daily  social  intercourse.  Face 
recognition  should  thus  be  perceived  as  non-intrusive.  For  the  same  reasons,  voiceprints  are  non- 
intrusive,  too.  Fingerprints,  however,  are  perceived  as  extremely  intrusive  because  of  their  association 
with  law  enforcement,  their  high  certainty  of  identification,  and  the  existence  of  extensive  databases  of 
fingerprints.  Retina  and  iris  identification  are  also  intrusive  because  they  provide  a  certainty  of 
identification  based  on  a  bodily  characteristic.  They  are  not  however,  associated  law  enforcement,  nor 
is  there  at  this  time  a  large  database  using  these  biometrics.  Hand  geometry  is  less  intrusive  as  it  has 
lower  certainty  of  identification. 

Performance 


Security 

This  criterion  has  been  subsumed  by  the  next  two  criteria,  and  its  weight  rolled  into  theirs. 

Probability  of  misidentification 

A  password  has  a  probability  of  misidentification  of  0,  as  do  the  card  technologies. 

Of  the  biometrics,  finger,  retina,  and  iris  have  a  probability  of  misidentification  that  is  essentially  0. 
Hand  geometry  also  has  a  very  low  level  of  misidentification. 

Signature  identification  reliability  is  dependent  on  the  identification  threshold  used.  This  is  true  of  voice 
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identification  also,  and  very  much  so  for  face  identification. 

Probability  of  non-identification 

A  password  has  a  probability  of  non-identification  of  0,  as  do  the  card  technologies. 

Of  the  biometrics,  finger,  retina,  and  iris  have  a  probability  of  non-identification  that  is  essentially  0. 
Hand  geometry  also  has  a  very  low  level  of  misidentification. 

Signature  identification  reliability  is  dependent  on  the  identification  threshold  used.  This  is  true  of  voice 
identification  also,  and  very  much  so  for  face  identification. 

Data  volume  required 

A  password  and  the  card  technologies  use  a  minimal  amount  of  data  for  identification,  on  the  order  of 
10  to  20  bytes. 

Finger,  retina,  and  iris  identification  use  more  data,  on  the  order  of  a  few  hundred  bytes. 

Hand  recognition  uses  a  black-and-white  image  of  the  hand,  32,000  pixels,  or  4,000  bytes. 

Voice  recognition  gathers  on  the  order  of  10  seconds  of  speech.  If  we  assume  that  statistics  are 
extracted  on  the  user’s  platform,  and  the  comparison  to  the  database  of  users  is  done  elsewhere,  the 
amount  of  data  to  be  transferred  to  the  identification  server  is  on  the  order  of  a  few  hundred  bytes. 

Face  recognition  uses  a  monochromatic  image  of  a  face.  If  we  assume  that  the  face  is  extracted  on  the 
user’s  platform,  and  the  comparison  to  the  database  of  users  is  done  elsewhere,  the  amount  of  data  to 
be  transferred  to  the  identification  server  is  on  the  order  of  1 ,000  to  2,000  bytes. 

Reliability 


Technology  maturity 

Password  identification  and  all  of  the  card  technologies  are  mature  technologies  that  have  been  fielded 
in  commercial  systems. 

Fingerprints  are  a  mature  technology.  Hand,  retina,  and  iris  identification  are  less  common. 

Voice  recognition  is  a  technology  that  is  approaching  maturity.  Face  recognition  is  a  still  emerging 
technology.  Its  installations  are  not  yet  industrial  grade. 

Reliability 

The  main  reliability  issue  is  the  sturdiness  of  the  readers. 

Password  can  rely  on  a  keyboard,  a  device  that  by  now  has  become  very  reliable.  Otherwise,  it  relies 
on  a  pen,  as  does  a  signature;  a  pen  and  its  screen  are  mature  technologies  and  are  highly  reliable. 

Of  the  card  technologies,  a  swipe  or  smart  card  reader  can  go  down  because  of  wear  and  dirt;  it  is  less 
reliable  than  a  proximity  card  because  it  relies  on  physical  contact. 

Hand,  iris,  retina  and  face  recognition  use  video  cameras,  which  are  reliable. 

Installed  base 
Passwords  are  ubiquitous. 

Swipe  cards  are  ubiquitous,  and  smart  cards  have  had  substantial  successful  installations,  for  example 
at  the  Olympics,  and  in  geographically  limited  commercial  installations.  Proximity  cards  are  in  wide 
use. 
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Fingerprint  readers  are  common,  while  retina,  iris,  and  hand  installations  are  as  yet  scanty. 

Voice  recognition  has  a  small  installed  base.  Face  recognition  is  new,  and  lacks  a  significant  installed 
base. 

Vendor  stability 

Vendors  of  password  technology  include  all  of  the  platform  manufacturers.  They  are  stable. 

The  current  major  developer  of  signature  technology  is  CIC.  There  are  resellers  of  their  technology. 
CIC  is  a  stable  company,  but  there  is  the  potential  for  instability  in  any  rapidly  advancing  technology. 
There  are  several  swipe  card  companies;  this  is  stable  industry.  Smart  card  vendors  are  also 
reasonably  stable.  Proximity  cards  are  less  common,  but  their  vendors  are  reasonably  stable. 
Fingerprint  recognition  vendors  are  stable.  Vendors  of  hand,  retina,  and  iris  identification  are  also 
reasonably  stable  companies,  but  these  again  are  vendors  in  a  narrow  market.  Only  one  stable 
company  currently  does  Iris  identification. 

Voice  recognition  is  available  from  different  vendors,  but  it  is  a  volatile  area.  Face  recognition  is 
available  from  essentially  one  vendor.  The  technology  is  so  new  that  the  long-term  prospects  of  the 
company  are  unclear. 

Cost 


Acquisition  cost 

All  of  the  technologies  require  some  kind  of  a  database  for  storing  the  user  profiles.  Some  of  the 
technologies  need  a  larger  database,  but  the  size  of  the  database  for  an  Air  Logistics  Center  is  not 
large  enough  that  the  difference  is  significant  in  terms  of  storage  cost. 

Password  cost  is  very  low;  all  platforms  come  with  the  technology  to  implement  them.  It  is  essentially 
free.  Implementation  requires  very  little  effort. 

All  of  the  means  of  biometric  identification  require  some  type  of  recognition  software.  The  price  of 
these  is  volatile  and  negotiable,  making  comparisons  difficult.  Card  recognition  requires  only  a  driver 
for  reading  the  card. 

Signature  technology  is  available  primarily  from  Communication  Intelligence  Corporation.  While  some 
implementations  require  a  signature  pad,  since  the  number  of  deployed  platforms  is  relatively  large,  it  is 
reasonable  to  assume  that  the  technology  to  use  in  an  Air  Logistics  Center  is  one  that  uses  the  built-in 
pen,  reducing  the  cost  only  to  software. 

Card  technology  requires  readers,  whose  cost  will  run  on  the  order  of  $30  each,  in  addition  to  licensing 
costs. 

Voice  recognition  requires  no  hardware. 

Face,  retina,  and  iris  recognition  all  require  some  form  of  video  input.  Iris  recognition  is  currently 
oriented  towards  ATM  applications,  with  specialized  video  hardware.  Video  cameras  in  a  housing  run 
on  the  order  of  $100  to  $200  in  small  quantities.  A  large  order  might  provide  them  at  a  cost  on  the 
order  of  $50.  These  require  integration  into  the  computing  platform,  since  very  few  platforms  include  a 
video  camera  as  a  standard  option. 

Hand  geometry  recognition  and  fingerprint  recognition  require  readers.  Hand  geometry  readers  are 
relatively  large,  costing  on  the  order  of  $100  per  reader,  while  fingerprint  readers  should  be  available  at 
about  $50.  Incorporation  or  attachment  of  fingerprint  readers  is  relatively  easy;  hand  readers  are  large, 
and  incorporation  would  be  expensive  and  difficult. 
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Maintenance  cost 


All  of  the  electronic  identification  means  share  some  maintenance  costs — management  of  the  user 
profile  database,  notably  enrollment  and  disenrollment.  There  are,  however,  differences  in  how 
operation  affects  costs. 

Maintenance  of  a  password  system  is  low.  Provision  must  be  made  for  changing  the  password 
periodically,  but  this  function  can  be  automated. 

Signature  recognition  requires  getting  sample  signatures  from  each  user.  Provision  must  be  made  for 
updating  the  signature;  again  this  can  be  automated. 

The  card  technologies  all  share  the  need  for  managing  the  physical  cards.  They  must  be  purchased 
and  distributed.  There  is  an  administrative  cost  to  accommodate  users  who  forget  their  cards.  Either 
“loaner"  cards  must  be  stocked  and  managed,  or  a  backup  identification  system  must  be  used,  say 
password. 

All  of  the  biometric  means  require  a  supervised  enrollment.  Hand,  finger,  retina,  and  iris  identification 
enrollment  can  be  done  just  once,  since  these  measurands  are  very  unlikely  to  change  over  time.  Only 
rare  events  such  as  loss  or  damage  of  a  finger  or  eye  will  change  a  user  profile.  Voices  and  faces, 
however,  are  apt  to  change  abruptly.  A  means  for  updating  them  must  be  provided.  The  Facelt 
package  has  the  capability  to  update  appearance  that  changes  gradually,  but  in  general  a  secondary 
means  of  identification  needs  to  be  supplied  for  temporary  changes  in  voice,  and  for  abrupt  changes  in 
appearance. 

Some  of  the  biometrics,  notably  signature  recognition,  voice  recognition,  and  face  recognition,  have 
threshold  values,  which  must  be  set.  A  cost  of  using  them  is  maintaining  a  person  knowledgeable 
enough  to  manage  the  threshold  so  as  to  ensure  efficient  and  accurate  identification. 

Administration 


Ease  of  installation 

When  an  electronic  identification  system  using  it  is  implemented,  the  majority  of  the  software  will  be 
incorporated  into  a  load  for  the  individual  workstations.  The  cost  for  installing  it  should  not  vary 
significantly  among  the  various  packages.  Similarly,  the  server  software  that  needs  to  be  installed  will 
be  roughly  comparable.  The  main  difference  will  then  be  the  cost  of  installing  hardware. 

Passwords  and  signatures  require  no  installation. 

All  of  the  card  technologies  require  installation  of  readers,  or  their  incorporation  into  the  platforms. 
Since  it  is  a  rare  platform  that  incorporates  such  a  reader,  this  is  a  significant  cost. 

Voice  requires  no  installation  if  the  platform  has  a  built-in  microphone,  a  peripheral  now  commonly 
found  in  portable  computers.  The  exception  might  be  for  older  engineering  workstations,  but  these  can 
be  easily  equipped  with  an  inexpensive  microphone. 

Face  recognition  requires  a  video  camera.  This  will  require  addition  of  a  camera  or  incorporation  of  a 
camera  into  the  platform.  Integration  is  apt  to  be  expensive,  since  an  integrated  camera  is  not  a 
common  option. 

Hand  recognition  requires  installation  of  hand  recognition  stations.  These  are  too  large  to  incorporate 
into  a  portable  computer. 

Fingerprint  recognition  hardware  will  be  available  at  a  cost  and  size  that  can  be  incorporated  into  a 
platform.  Since  commercial  platforms  are  unlikely  to  include  these,  this  would  be  a  custom  installation, 
with  the  accompanying  cost. 

Retina  and  iris  recognition  packages  currently  available  are  not  set  up  to  incorporate  into  a  portable 
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computer.  The  technology  could,  however,  be  so  incorporated,  requiring,  like  face  recognition,  a 
camera  as  part  of  the  portable  platform. 

Ease  of  management 

In  all  cases,  there  is  a  need  to  manage  the  user  profile  database  that  is  roughly  uniform  no  matter  what 
means  of  electronic  identification  is  used.  The  main  tasks  here  are  enrollment  and  disenrollment  of 
users. 

Passwords  need  to  be  changed  periodically.  This  can  be  automated,  so  that  no  management  need  be 
done. 

Signatures  may  change,  due  to  name  changes,  or  due  to  changes  in  how  an  individual  writes. 
Provision  for  updating  a  user’s  profile  can  be  automated,  so  that  no  management  need  be  done. 
Signature  recognition  requires  management  of  the  recognition  threshold. 

Electronic  identification  using  cards  requires  managing  the  physical  cards.  Provision  must  be  made  for 
replacing  lost  or  damaged  cards,  and  for  “loaners"  for  forgotten  cards. 

Voice  recognition  requires  a  provision  for  days  when  a  voice  changes,  probably  through  the  provision  of 
a  secondary  means  of  identification,  for  example  password.  If  password  recognition  is  used,  no 
management  is  required  beyond  the  effort  required  to  maintain  a  separate  means  of  identification. 
Voice  recognition  requires  management  of  the  recognition  threshold,  both  for  extracting  the  face  as  well 
as  recognizing  the  face. 

Face  recognition  requires  a  provision  for  updating  the  user’s  template  in  the  database  when  the  user’s 
face  changes.  The  user  can  do  this,  if  a  secondary  means  of  identification  is  provided  to  ensure  that 
the  updated  template  is  indeed  that  of  the  user.  Face  recognition  requires  management  of  the 
recognition  threshold. 

Hand,  finger,  retina,  and  iris  identification  require  little  management,  once  they  have  been  installed. 

4.0  Product  Experience 

Voice  Print,  a  voice  recognition  package  was  installed  at  ATL  and  exercised  there.  The  Facelt  package 
was  installed  and  exercised.  A  version  of  Voice  Print  that  integrated  voice  and  face  recognition  was 
installed  and  exercised.  The  SignOn  Systems  SignOn  Verify  signature  recognition  package  was 
installed  on  a  Telxon  1134  using  Windows  for  Pen  Computing.  A  smart  card  was  exercised. 
Technologies  in  common  use  (e.g.,  password  and  swipe  or  proximity  card  identification)  were  not 
exercised,  as  they  are  mature  and  are  familiar  to  Air  Logistics  Center  users. 


5.0  Source  Summary 

The  information  on  which  the  evaluation  was  based  was  drawn  from  a  variety  of  sources.  Material  from 
the  Web  sites  of  various  vendors  provided  much  technical  information  about  the  packages  under 
evaluation.  Journals  were  consulted,  notably  Pen  Computing,  but  also  Infoworld,  PC  magazine  and 
other  trade  journals.  Adventitious  information  encountered  in  newspapers  such  as  the  Wall  Street 
Journal  was  also  used.  Vendors’  sales  organizations  and  technical  personnel  were  also  contacted. 
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Supplement  A-3-1  Electronic  Identification  Trade  Study  Criteria 


CRITERIA 

| 

13 

COMMENTS 

1  Capabilities 

Time  to  identify 

3 

This  time  includes  the  time  at  the  reader  and  the 
database.  It  does  not  include  LAN  time. 

1  >10  seconds 

10  <1  second 

Weight  of  medium 

2 

1  >  2  oz 

10  weightless 

Weight  of  reader 

2 

1  >  4  oz 

10  weightless 

Size  of  medium 

2 

1  Barely  fits  in  a  shirt  pocket 

10  No  medium 

Size  of  reader 

2 

1  Not  portable 

10  Integrated  with  any  portable  computer 

Power  consumption  of  reader 

2 

This  is  power  consumption  attributable  to  the 
electronic  ID  system  alone  when  used  with 
portable  devices. 

1  2: 1  amp-hour  in  daily  cycle 

10  £  0.1  amp-hour  in  daily  cycle. 

Robustness  in  hangar  environment 

2 

The  hangar  has  heat  swings,  poor  lighting,  and 
can  be  noisy. 

1  Performs  poorly  in  weak  lighting  and  is 
temperature  sensitive 

10  Is  unaffected. 

Ease  of  Use 

Learning  curve 

2 

1  The  user  interface  is  complex  and  requires  a 
half  a  day  training  to  use  effectively. 

10  Interfaces  are  intuitive  and  can  be  mastered 
in  5  minutes 

Convenience 

6 

1  Electronic  identification  requires  going  to  a 
workstation  outside  the  aircraft 

10  Electronic  identification  can  be  done  in 
cramped  quarters 

Intrusiveness 

10 

1  Electronic  identification  uses  biometric 

measurements  and  uses  an  active  device  on 
sensitive  organs 

10  Electronic  identification  is  psychologically 
and  physically  not  intrusive. 
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Supplement  A-3-1  Electronic  Identification  Trade  Study  Criteria  (Continued) 


Performance 

Security 

0 

1  System  can  be  compromised  by 
unsophisticated  users 

10  System  can  be  compromised  by 
administrator  only  with  difficulty 

Probability  of  misidentification 

4 

1  >.1 

10  <.001 

Probability  of  non-identification 

4 

1  >.2 

10  <.01 

Data  volume  required  for 
identification 

2 

1  >10,000  bytes 

10  <20  bytes 

Reliability 

Product  maturity 

4 

1  In  beta 

10  In  wide  use  5  years 

Reliability 

3 

1  Most  users  report  problems 

10  No  known  problems 

Installed  base 

3 

1  Single  user  base 

10  500  or  more  users 

Vendor  stability 

4 

1  Vendor  in  business  less  than  a  year 

10  Vendor  in  business  5  years  or  more 

Cost 

Acquisition  cost 

3 

This  is  for  a  fully  fielded  system  with  1000 
users. 

1  Over  $200K 

10  Free 

Maintenance  cost 

4 

This  is  the  cost  of  annual  licenses  and 
upgrades 

1  Over  $50K 

10  Free 

Administration 

Ease  of  installation 

3 

1  One  week  or  more  per  platform 

1 0  One  day  or  less  per  platform 

Ease  of  management 

4 

1  Requires  full  time  administrator 

10  Once  installed,  needs  no  administration 
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Supplement  A-3-2  Criterion  Weight  Algorithm 


Consider  N  criterion  groups,  the  /th  with  weight  Wj. 

m 

Let  the  /th  group  have  nt  criteria,  the  yth  with  weight  w».  Let  oi-X  •  Let  M  be  the  highest  possible 

>1 

score. 

Let  a  product  score  s,y  on  the  yth  criterion  in  the  /th  group.  The  total  product  score  is  then 


The  result  is  to  weight  each  criterion  by 


Wi  Wij 


Oi 


max 

Finally,  find  the  biggest  weight  =  .  . 

hJ 


fWiwA 


y  at  j 


,  call  it  n,  and  normalize  so  that  the  corresponding 


criterion  has  weight  10. 

Thus  weight  the  yth  criterion  in  the  /th  group  by 


10  Wt  Wij 
H  Oi 

and  round  to  an  integer  value. 
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This  spreadsheet  computes  overall  criterion  weights  from  criterion  group  weights 
and  criterion  weights  within  each  criterion  group  according  to  the  algorithm  above. 

It  is  the  source  of  evaluation  criterion  weights  from  Table  A-3-1. 

max  weight  for  normalization:  1 0 


Criterion 

intra¬ 

group 

raw 

criterion 

normalized 

Groups  Criteria  Criterion  qroup  weights 

weights 

weights 

weights 

Capabilities  10 

Language  compatibility 

5 

1.613 

5 

Platform  compatibility 

5 

1.613 

5 

Supporting  product  availability 

4 

1.290 

4 

Standards  based 

5 

1.613 

5 

Flexibility 

5 

1.613 

5 

Distribution 

4 

1.290 

4 

Scalability 

3 

0.968 

3 

31 

Ease  of  Use  4 

Learning  curve 

5 

2.000 

7 

Implementation  ease 

5 

2.000 

7 

10 

Performance  8 

Throughput  rate 

4 

1.600 

5 

Time  to  establish  connection 

4 

1.600 

5 

Maximum  number  of  connections 

4 

1.600 

5 

Time  to  bring  up  applications 

4 

1.600 

5 

Memory  needed 

4 

1.600 

5 

20 

Reliability  8 

Product  maturity 

4 

2.286 

8 

Reliability 

3 

1.714 

6 

installed  base 

3 

1.714 

6 

Vendor  stability 

4 

2.286 

8 

14 

Cost  4 

Acquisition  cost 

4 

1.600 

5 

Maintenance  cost 

6 

2.400 

8 

10 

Administration  6 

Ease  of  installation 

3 

3.000 

10 

Ease  of  management 

3 

3.000 

10 

6 

max 

3.000 
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Supplement  A-3-3  Companies 


Communication  Intelligence  Corporation 
275  Shoreline  Drive 
Redwood  Shores,  CA  94065 

Recognition  Systems,  Inc. 

1520  Dell  Avenue 
Campbell,  CA  95008 

Identitech 

100  Rialto  Place 

Melbourne  FL  32901 

Annasoft  Systems 
11838  Bernardo  Plaza  Court 
San  Diego  CA  92128 

Recognition  Systems,  Inc. 

1520  Dell  Avenue 
Campbell  CA  95008 

Keyware  Technologies 
Excelsiorlaan  28-30 
B-1930  Zaventem 
Belgium 

500  West  Cummings  Park 
Woburn  MA  01801 

Visionics  Corporation 
1  Exchange  Place 
Suite  810 

Jersey  City,  NJ  07302 

App  Informatik  Davos 
PO  Box  185 
7260  Davos-Dorf 
Switzerland 

Vasco  Data  Security 
1919  South  Highland  Avenue 
Suite  1180C 
Lombard,  IL  60148 
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Sensar,  Incorporated 
121  Whittendale  Drive 
Moorestown,  NJ 

Omron 

Karasuma  Street  Chichijo  Sagaru 

Kyoto-city  Kyoto  600 

Japan 

Security  Dynamics 
20  Crosby  Drive 
Bedford  MA  0173o 

Miros,  Incorporated 
Suite  18 

572  Washington  St 
Wellesley,  MA  02181 

Nordra  Technologies 
PO  Box  645 
Andover  NJ  07821 


A-4.  Energy  Sources  Trade  Study 
Introduction 

A  survey  of  available  battery  technology  was  conducted  using  both  manufacturers  and 
experimental  data.  Four  common  battery  chemistries  were  included  in  the  survey  of 
technology:  Lithium  Ion  (Table  A-4-1),  Zinc  Air  (Table  A-4-2),  Nickel  Cadmium  (Table  A-4-3), 
and  Nickel  Metal-Hydride  (Table  A-4-4).  Our  purpose  was  to  evaluate  the  currently  available 
technology  for  use  in  wearable  computing  systems.  The  target  system  we  used  require  12 
Volts  DC,  drew  3  amperes,  and  was  required  to  operate  at  least  4  hours  before  changing  the 
battery  pack. 

Each  type  of  battery  was  rated  based  on  the  composite  information  of  the  group  as  a  whole. 
The  evaluation  area  were: 

•  Weight 

•  Capacity 

•  Specific  Energy 

•  Form  Factor 

•  Voltage 

•  Recharge  Time 

•  Battery  Leakage 

•  System  Standby  Power  Consumption 

•  Shelf  Life 

•  Operating  Temperatures 

•  Transportability 

•  Safety 

•  Disposability 

•  Logistics 

•  Cost 

Initially,  we  had  planned  for  a  144-watt  battery  pack.  After  the  application  began  to  take  shape 
and  undergo  preliminary  testing,  it  became  clear  that  half  that  number  would  suffice.  It  is  our 
hope  that  during  the  final  round  of  testing  and  evaluation  we  will  be  able  to  reduce  this  number 
once  again  by  accurately  determining  the  maximum  reasonable  length  for  a  computer-based 
video  conferencing  session.  Cost  data  was  collected  from  a  range  of  distributors  of  various 
manufacturers. 
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Analysis 


Our  analysis  indicates  that  the  top  two  contenders  for  power  solution  are  zinc  air,  and  lithium 
Ion,  based  on  energy  capacity  and  mass.  The  zinc  air  cells  reviewed  in  this  study  were  very 
light,  and  had  a  high  energy  density.  Unfortunately,  they  were  designed  for  low  current 
applications,  and  are  incapable  of  supplying  the  current  required  based  on  our  current 
application  data.  This  current  limit  is  related  to  the  replacement  rate  of  oxygen  in  the  cell.  The 
cells  can  become  oxygen  starved  during  high  current  output  due  to  the  rapid  consumption  of 
available  oxygen  by  the  electrochemical  process.  Zinc  Air  cells  can  be  used  in  higher  current 
applications,  but  higher  current  cells  required  more  air  pass  through  in  the  cells  than  the  low 
current  cells.  Since  this  would  require  a  large  airflow  from  the  environment  into  the  battery,  it 
is  likely  that  the  cell  would  become  contaminated,  and  fail  prematurely. 

Though  much  less  expensive,  and  recyclable,  the  NI-CD  solutions  are  significantly  heavier 
than  the  Li-Ion  cells,  as  are  the  NI-MH  units.  Further,  the  lithium  cells  do  not  suffer  from  the 
“memory”  problem  NI-CD  units  are  prone  to.  Li-Ion  packs  are  typically  rated  for  a  number  of 
cycles  similar  to  other  technologies,  about  750.  This  is  dependant  on  its  use  profile  and  depth 
of  discharges. 


Model  Limitations  and  Conclusions 

The  lithium  ion  cells  are  clearly  more  expensive  than  the  competing  technologies.  After 
eliminating  the  zinc  air  cells,  the  least  massive  competitor  for  the  lithium  ion  unit  is  nickel 
cadmium,  which  masses  2.38  Ibm,  which  the  lithium  pack  is  nearly  40%  less  massive  at  1.48 
Ibm.  There  are  lighter  lithium  ion  solutions,  but  those  cells  cannot  handle  the  continuous 
current  rate  required  by  our  application. 

There  is  a  large  tradeoff  to  be  made  between  cost  and  mass.  For  this  type  of  application  it  is 
our  opinion  that  mass  should  be  the  dominant  factor,  given  that  the  cost  of  the  lithium  pack 
would  only  represent  less  than  10%  of  the  expenditure  for  the  processing  unit  itself.  Further, 
the  use  profile  of  the  units  has  yet  to  be  completely.  It  is  as  yet  unclear  what  percentage  of 
time  the  unit  will  be  using  its  full  functionality.  During  field-testing,  we  will  use  aggressive 
power  management  techniques,  and  further  refine  our  energy  model.  This  may  reduce  our 
energy  requirements  farther  to  a  point  where  other  technologies  can  be  considered,  and  the 
cost  of  a  lithium  pack  will  be  reduced  yet  again. 
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Table  A-4-1  Energy  Sources  Trade  Study:  Lithium  Ion 


Criteria 

Rqmts 

Weight 

Scor 

e 

Tota 

1 

Comments 

1.0  Performance 

Weight  -  The  battery  should  be 
less  than  half  of  the  total  system 
weight  and  not  more  than  1  pound 

Derived 

311011 

10 

10 

100 

Mass  of  battery  slightly 
more  than  target,  but 
lightest  meeting  current 
model  specification.  As 
the  model  improves,  the 
mass  should  decrease  to 
meet  spec. 

Capacity  -  The  system  should  be 
able  to  run  for  the  duration  of  a 
job  or  have  hot  swap  ability. 
Amperage  requirements  for  a 
wearable  computer  typically  range 
from  3-7  amps,  so  for  example,  for 
a  4-hour  job;  capacity  for  a  battery 
needs  to  be  between  12-28  amp 
hours.  Batteries  typically  range 
from  50  -  2400  milliamp-hours 
therefore  a  battery  pack  will  need 
to  be  used. 

Derived 

311011 

10 

10 

100 

Systems  capable  of 
meeting  capacity 
specification. 
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j  Table  A-4-1  Energy  Sources  Trade  Study:  Lithium  Ion  (Continued) 

Specific  Energy  -  The  weight 
(W)  requirements,  the  capacity 
(C)  requirements,  and  the  voltage 
(V)  requirements  detennine  the 
specific  energy  (SE)  requirements 
by  the  equation  SE=V*C/W. 
Typical  specific  energies  for 
batteries  range  from  35  to  260.  In 
order  of  lowest  to  highest  specific 
energy,  typical  chemistries  with 
their  specific  energies  are  NiCad 
(35),  Lead-acid  (35), 
Zn/alkaline/MnO  (95),  Li-ion 
(140),  Zn-Air  (220). 

Derived 

311011 

10 

8 

Specific  energy  is 
higher  than  most 
systems,  but  not  as  good 
as  the  zinc-air 
technologies. 

Form  Factor  -  standard  form 
factors  are  AAA,  AA,  C,  D,  etc. 
The  battery  chemistry  and  the 
weight  together  determine  the 
volume  needed  for  the  battery. 
Factors  to  consider  include  ease  of 
replacement  and  packing  density. 

Derived 

311011 

10 

10 

100 

Available  in  a  wide 
variety  of  form  factors. 

Voltage  -  The  voltage  should  be 
compatible  with  the  dominant 
voltage  of  the  system  components. 
Step  up  or  step  down  DC-to-DC 
converters  can  be  used  for  the 
other  components.  Typical 
voltages  for  wearable  computer 
components  range  from  3-12  volts 

Derived 

311011 

10 

10 

100 

Capable  of  building  cell 
pack  to  deliver  required 
voltage. 
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Table  A-4-1  Enerev  Sources  Trade  Study:  Lithium  Ion  (Continued 


9 


Recharge  time  -  This  must  be  less 
than  the  duty  cycle  of  the  battery 
divided  by  the  number  of  battery 
sets  to  insure  ready  availability. 

For  battery  chemistries  that  exhibit 
memory,  the  recharge  time  will  be 
lengthened  by  a  mandatory 
complete  discharge  prior  to 
commencement  of  charging. 

Derived 

311012 

Battery  Leakage  - 

This  is  the  discharge  rate  under  no 
load  conditions.  The  lower  the  rate 
the  better  -  a  goal  should  be  a  no 
load  discharge  rate  of  1000  times 
less  than  when  fully  operational. 

Derived 

311011 

System  Standby  Power 
Consumption  (SSPC)  (i.e.  sleep 
mode)  -  should  be  100  times 
longer  than  operational  time.  E.g. 
if  operational  time  is  2  hours,  then 
SSPC  should  be  roughly  a  week. 

Derived  1 
311011 

This  is  the  rate  at  which 
the  battery  loses  power 
in  the  system  when  it  is 
not  being  used. 


Shelf  Life  -  at  least  1  year 
within  temperature  range  -30C  to 
50C 

Derived 

311011 

5 

3 

15 

Meets  requirement 

Operating  Temperatures  -  at 
minimum  should  operate  with  - 
20C  -  +40C  with  negligible 
performance  degradation 

Derived 

311011 

10 

3 

30 

Meets  Requirement 

Transportation  -  Are  there  any 
restrictions  on  modes  of 
transportation? 

Derived 

311022 

6 

6 

36 

Transport  as  hazardous 
material. 
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[  Table  A-4-1  Energy  Sources  Trade  Study:  Lithium  Ion  (Continued) 

Safety  - 

1)  if  battery  is  shorted  will  it  -  a) 
rupture?  b)  explode? 

2)  Does  it  vent  toxic  gases 

3)  Does  it  leak  its  contents  over 
time? 

Derived 

311022 

8 

8 

10 

Leaks  electrolyte  if 
punctured 

3.0  Maintainability 

Disposability  - 

a)  recyclable 

b)  landfill  safe 

Derived 

301201 

8 

5 

40 

Must  dispose  of  in 
hazmat  landfill. 

4.0  Producibility 

Logistics  -  Are  the  batteries 
readily  available?  (Are  they  a 
commodity  product  available  from 
several  sources?) 

Derived 

311011 

9 

10 

90 

Available  from  multiple 
manufacturers. 

Cost  -  For  greatest  economy, 
minimize  the  cost  per  operational 
usage  hour.  I.e.  cost  divided  by  the 
operating  time,  where  maximum 
operating  time  =  (cyclability  * 
(capacity/amps).  Cyclability 
typically  ranges  from  1  (for  a  non- 
rechargeable  battery)  to  1000. 

Derived 

311011 

8 

8 

64 

-$400.00 

Total: 

1001 

This  trade  study  is  for  energy  sources  for  the  ITI-ALC  Maintenance  Support  Device  (MSD).  The  goal 
for  the  MSD  is  sufficient  capacity  to  operate  a  complete  8-hour  shift  with  at  most  one  battery  change. 
For  example,  the  system  duty  cycle  for  an  inspection  task  may  range  from  as  low  as  20%  to  as  high  as 
100%  but  will  be  a  function  of  the  logistics  activity  (which  is  TBD).  Similar  variations  are  possible  with 
other  task  types.  The  trade  study  criteria  define  a  decision  procedure  for  choosing  a  battery  once  the 
weight,  dominant  voltage,  and  needed  capacity  of  the  overall  system  is  known.  Hence,  several  criteria 
are  described  by  equations  rather  than  absolute  ranges  of  values.  Once  these  parameters  are  known 
with  greater  precision  (from  the  concurrent  work  on  the  wearable  computer  trade  studies),  we  will  be 
able  to  choose/design  the  battery  pack  with  greater  precision. 
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Table  A-4-2  Energy  Sources  Trade  Study:  Zinc  Air 


Criteria 

Score 

Total 

Comments 

1.0  Performance 

Weight  -  The  battery  should  be 
less  than  half  of  the  total  system 
weight  and  not  more  than  1  pound 

Derived 

311011 

10 

10 

100 

least  massive 

Capacity  -  The  system  should  be 
able  to  run  for  the  duration  of  a 
job  or  have  hot  swap  ability. 
Amperage  requirements  for  a 
wearable  computer  typically  range 
from  3-7  amps,  so  for  example,  for 
a  4-hour  job;  capacity  for  a  battery 
needs  to  be  between  12-28  amp 
hours.  Batteries  typically  range 
from  50  -  2400  milliamp-hours 
therefore  a  battery  pack  will  need 
to  be  used. 

Derived 

311011 

10 

10 

100 

Can  store  required 
energy 

Specific  Energy  -  The  weight 
(W)  requirements,  the  capacity 
(C)  requirements,  and  the  voltage 
(V)  requirements  determine  the 
specific  energy  (SE)  requirements 
by  the  equation  SE=V*C/W. 
Typical  specific  energies  for 
batteries  range  from  35  to  260.  In 
order  of  lowest  to  highest  specific 
energy,  typical  chemistries  with 
their  specific  energies  are  NiCad 
(35),  Lead-acid  (35), 
Zn/alkaline/MnO  (95),  Li-ion 
(140),  Zn-Air  (220). 

Derived 

311011 

10 

10 

100 

Highest  SE  of  any 

evaluated 

technology. 

Table  A-4-2  Energy  Sources  Trade  Study:  Zinc  Air  (Continued) 


Form  Factor  -  standard  form 
factors  are  AAA,  AA,  C,  D,  etc. 
The  battery  chemistry  and  the 
weight  together  determine  the 
volume  needed  for  the  battery. 
Factors  to  consider  include  ease  of 
replacement  and  packing  density. 

Derived 

311011 

10 

7 

70 

Least  massive  cells 
are  coin  type,  and 
would  require  large 
numbers 

Voltage  -  The  voltage  should  be 
compatible  with  the  dominant 
voltage  of  the  system  components. 
Step  up  or  step  down  DC-to-DC 
converters  can  be  used  for  the 
other  components.  Typical 
voltages  for  wearable  computer 
components  range  from  3-12  volts 

Derived 

311011 

10 

10 

100 

Capable  of  building 
cell  pack  to  deliver 
required  voltage. 

Recharge  time  -  This  must  be  less 
than  the  duty  cycle  of  the  battery 
divided  by  the  number  of  battery 
sets  to  insure  ready  availability. 

For  battery  chemistries  that  exhibit 
memory,  the  recharge  time  will  be 
lengthened  by  a  mandatory 
complete  discharge  prior  to 
commencement  of  charging. 

Derived 

311012 

8 

1 

8 

Rated  poorly, 
because  the  pack  will 
need  16  hours  to 
charge,  necessitating 
multiple  packs  per 
unit.  Also,  some 
models  not 
rechargeable. 

Battery  Leakage  - 

This  is  the  discharge  rate  under  no 
load  conditions.  The  lower  the  rate 
the  better  -  a  goal  should  be  a  no 
load  discharge  rate  of  1 000  times 
less  than  when  fully  operational. 

Derived 

311011 

6 

10 

60 

Extremely  low 
leakage. 
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Table  A-4-2  Energy  Sources  Trade  Study:  Zinc  Air  (Continued) 


System  Standby  Power 

Derived 

Consumption  (SSPC)  (i.e.  sleep 

311011 

mode)  -  should  be  100  times 

longer  than  operational  time.  E.g. 

if  operational  time  is  2  hours,  then 

SSPC  should  be  roughly  a  week. 

2.0  Reliabili 


Shelf  Life  -  at  least  1  year  Derived 

within  temperature  range  -3  0C  to  311011 
50C 


Operating  Temperatures -at  Derived 

minimum  should  operate  with  -  311011 

20C  -  +40C  with  negligible 
erformance  degradation 


Transportation  -  Are  there  any  Derived 
restrictions  on  modes  of  311 022 

transportation? 


Safety  -  Derived 

1 )  if  battery  is  shorted  will  it  -  a)  311 022 
rupture?  b)  explode? 

2)  Does  it  vent  toxic  gases 

3)  Does  it  leak  its  contents  over 
time? 


3.0  Maintainabili 


Disposability  -  Derived 

a)  recyclable  301201 

b)  landfill  safe 


4.0  Producibili 


Logistics  -  Are  the  batteries  Derived 

readily  available?  (Are  they  a  311011 
commodity  product  available  from 
several  sources?) 


Meets  specification. 


Meets  Specification 


Meets  Specification 


No  Restrictions  on 
transportation. 


Does  not  pose  a  hazard. 


No  disposal  restrictions. 


Available  from  several 
manufacturers 
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|  Table  A-4-2  Energy  Sources  Trade  Study:  Zinc  Air  ( 

Continued) 

Cost  -  For  greatest  economy, 
minimize  the  cost  per  operational 
usage  hour.  I.e.  cost  divided  by  the 
operating  time,  where  maximum 
operating  time  =  (cyclability  * 
(capacity/amps).  Cyclability 
typically  ranges  from  1  (for  a  non- 
rechargeable  battery)  to  1000. 

Derived 

311011 

8 

8 

64 

-400-500 

Total: 

1003 

This  trade  study  is  for  energy  sources  for  the  ITI-ALC  Maintenance  Support  Device  (MSD).  The  goal 
for  the  MSD  is  sufficient  capacity  to  operate  a  complete  8-hour  shift  with  at  most  one  battery  change. 
For  example,  the  system  duty  cycle  for  an  inspection  task  may  range  from  as  low  as  20%  to  as  high  as 
100%  but  will  be  a  function  of  the  logistics  activity  (which  is  TBD).  Similar  variations  are  possible  with 
other  task  types.  The  trade  study  criteria  define  a  decision  procedure  for  choosing  a  battery  once  the 
weight,  dominant  voltage,  and  needed  capacity  of  the  overall  system  is  known.  Hence,  several  criteria 
are  described  by  equations  rather  than  absolute  ranges  of  values.  Once  these  parameters  are  known 
with  greater  precision  (from  the  concurrent  work  on  the  wearable  computer  trade  studies),  we  will  be 
able  to  choose/design  the  battery  pack  with  greater  precision. 
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Table  A-4-3  Energy  Sources  Trade  Study:  Nickel  Cadmium 


Criteria _ Rqmts  Weight  Score  Total  Comments _ 

1.0  Performance _ . _ 

Weight  -  The  battery  should  be  Derived  10  0  0  Packs  made  of  this 

less  than  half  of  the  total  system  311011  technology  will  not 

weight  and  not  more  than  1  pound  meet  specified  mass 

_ _ requirements. _ 

Capacity  -  The  system  should  be  Derived  10  10  100  Capable  of  holding  the 

able  to  run  for  the  duration  ofa  311011  required  energy, 

job  or  have  hot  swap  ability. 

Amperage  requirements  for  a 
wearable  computer  typically  range 
from  3-7  amps,  so  for  example,  for 
a  4-hour  job;  capacity  for  a  battery 
needs  to  be  between  12-28  amp 
hours.  Batteries  typically  range 
from  50  -  2400  milliamp-hours 
therefore  a  battery  pack  will  need 

to  be  used. _ 

Specific  Energy  -  The  weight  Derived  10  2  20  Extremely  low. 

(W)  requirements,  the  capacity  311011 
(C)  requirements,  and  the  voltage 
(V)  requirements  determine  the 
specific  energy  (SE)  requirements 
by  the  equation  SE=V*C/W. 

Typical  specific  energies  for 
batteries  range  from  35  to  260.  In 
order  of  lowest  to  highest  specific 
energy,  typical  chemistries  with 
their  specific  energies  are  NiCad 
(35),  Lead-acid  (35), 

Zn/alkaline/MnO  (95),  Li-ion 

(140),  Zn-Air  (220). _ | _ | _  1  1 _ 
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Table  A-4-3  Energy  Sources  Trade  Study:  Nickel  Cadmium  (Continued) 


Form  Factor  -  standard  form 
factors  are  AAA,  AA,  C,  D,  etc. 
The  battery  chemistry  and  the 
weight  together  determine  the 
volume  needed  for  the  battery. 
Factors  to  consider  include  ease  of 
replacement  and  packing  density. 

Derived 

311011 

10 

2 

20 

Cells  in  many  shapes, 
but  too  bulky  in  large 
enough  numbers  to 
meet  energy 
requirements 

Voltage  -  The  voltage  should  be 
compatible  with  the  dominant 
voltage  of  the  system  components. 
Step  up  or  step  down  DC-to-DC 
converters  can  be  used  for  the 
other  components.  Typical 
voltages  for  wearable  computer 
components  range  from  3-12  volts 

Derived 

311011 

10 

10 

100 

Capable  of  building 
cell  pack  to  deliver 
required  voltage. 

Recharge  time  -  This  must  be  less 
than  the  duty  cycle  of  the  battery 
divided  by  the  number  of  battery 
sets  to  insure  ready  availability. 

For  battery  chemistries  that  exhibit 
memory,  the  recharge  time  will  be 
lengthened  by  a  mandatory 
complete  discharge  prior  to 
commencement  of  charging. 

Derived 

311012 

8 

TO 

80 

Can  fast  charge  in  1 .5 
hours.  By  far  the  best 
technology  for  quick 
recharge. 

Battery  Leakage  - 

This  is  the  discharge  rate  under  no 
load  conditions.  The  lower  the  rate 
the  better  -  a  goal  should  be  a  no 
load  discharge  rate  of  1 000  times 
less  than  when  fully  operational. 

Derived 

311011 

6 

3 

18 

Higher  than  average 
leakage  rate 
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Table  A-4-3  Energy  Sources  Trade  Study:  Nickel  Cadmium  (Continued 


10  70  Meets  requirement 


System  Standby  Power 

Derived 

Consumption  (SSPC)  (i.e.  sleep 

311011 

mode)  -  should  be  100  times 

longer  than  operational  time.  E.g. 

if  operational  time  is  2  hours,  then 

SSPC  should  be  roughly  a  week. 

2.0  Reliabili 


Shelf  Life  -  at  least  1  year 
within  temperature  range  -30C  to 
50C 

Derived 

311011 

5 

3 

15 

Meets  specification 

Operating  Temperatures  -  at 
minimum  should  operate  with  - 
20C  -  +40C  with  negligible 
performance  degradation 

Derived 

311011 

10 

3 

30 

Meets  specification 

Transportation  -  Are  there  any 
restrictions  on  modes  of 
transportation? 

Derived 

311022 

6 

10 

60 

No  restrictions  on 
transportation. 

Safety  - 

1)  if  battery  is  shorted  will  it  -  a) 
rupture?  b)  explode? 

2)  Does  it  vent  toxic  gases 

3)  Does  it  leak  its  contents  over 
time? 

Derived 

311022 

8 

8 

64 

Can  leak  after 
extended  use. 

3.0  Maintainabili 


Disposability  - 

a)  recyclable 

b)  landfill  safe 


4.0  Producibili 


Logistics  -  Are  the  batteries 
readily  available?  (Are  they  a 
commodity  product  available  from 
several  sources?) 


Derived  8 
301201 


Derived 

311011 


Recyclable.  Only 
chemistry,  which  can 
be  recycled. 


Available  from 
several  manufacturers. 
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Table  A-4-3  Energy  Sources  Trade  Study:  Nickel  Cadmium  (Continued)  | 

Cost  -  For  greatest  economy, 
minimize  the  cost  per  operational 
usage  hour.  I.e.  cost  divided  by  the 
operating  time,  where  maximum 
operating  time  =  (cyclability  * 
(capacity/amps).  Cyclability 
typically  ranges  from  1  (for  a  non- 
rechargeable  battery)  to  1000. 

Derived 

311011 

8 

10 

80 

-$250.00 

Total: 

835 

This  trade  study  is  for  energy  sources  for  the  ITI-ALC  Maintenance  Support  Device  (MSD).  The  goal 
for  the  MSD  is  sufficient  capacity  to  operate  a  complete  8-hour  shift  with  at  most  one  battery  change. 
For  example,  the  system  duty  cycle  for  an  inspection  task  may  range  from  as  low  as  20%  to  as  high  as 
100%  but  will  be  a  function  of  the  logistics  activity  (which  is  TBD).  Similar  variations  are  possible  with 
other  task  types.  The  trade  study  criteria  define  a  decision  procedure  for  choosing  a  battery  once  the 
weight,  dominant  voltage,  and  needed  capacity  of  the  overall  system  is  known.  Hence,  several  criteria 
are  described  by  equations  rather  than  absolute  ranges  of  values.  Once  these  parameters  are  known 
with  greater  precision  (from  the  concurrent  work  on  the  wearable  computer  trade  studies),  we  will  be 
able  to  choose/design  the  battery  pack  with  greater  precision. 
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Table  A-4-4  Energy  Sources  Trade  Study:  Nickel  Metal-Hydride 


Criteria _ Rqmts  Weight  Score  Total  Comments _ 

1.0  Performance _ 

Weight  -  The  battery  should  be  Derived  10  0  0  This  technology 

less  than  half  of  the  total  system  311011  cannot  meet  mass 

weight  and  not  more  than  1  pound  specification. 

Capacity  -  The  system  should  be  Derived  10  10  100  Capable  of  holding 

able  to  run  for  the  duration  of  a  311011  required  amount  of 

job  or  have  hot  swap  ability.  energy. 

Amperage  requirements  for  a 
wearable  computer  typically  range 
from  3-7  amps,  so  for  example,  for 
a  4-hour  job;  capacity  for  a  battery 
needs  to  be  between  12-28  amp 
hours.  Batteries  typically  range 
from  50  -  2400  milliamp-hours 
therefore  a  battery  pack  will  need 

to  be  used. _ 

Specific  Energy  -  The  weight  Derived  10  2  20  Extremely  low. 

( W)  requirements,  the  capacity  311011 
(C)  requirements,  and  the  voltage 
(V)  requirements  determine  the 
specific  energy  (SE)  requirements 
by  the  equation  SE=V*C/W. 

Typical  specific  energies  for 
batteries  range  from  35  to  260.  In 
order  of  lowest  to  highest  specific 
energy,  typical  chemistries  with 
their  specific  energies  are  NiCad 
(35),  Lead-acid  (35), 

Zn/alkaline/MnO  (95),  Li-ion 

(140),  Zn-Air  (220).  _ ^ _ | _  1 _ 
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Table  A-4-4  Energy  Sources  Trade  Study:  Nickel  Metal-Hydride  (Continued) 


Form  Factor  -  standard  form  Derived 
factors  are  AAA,  AA,  C,  D,  etc.  311011 
The  battery  chemistry  and  the 
weight  together  determine  the 
volume  needed  for  the  battery. 

Factors  to  consider  include  ease  of 
replacement  and  packing  density. _ 


Voltage  -  The  voltage  should  be  Derived 
compatible  with  the  dominant  311011 
voltage  of  the  system  components. 

Step  up  or  step  down  DC-to-DC 
converters  can  be  used  for  the 
other  components.  Typical 
voltages  for  wearable  computer 
components  range  from  3-12  volts 


Recharge  time  -  This  must  be  less  Derived 
than  the  duty  cycle  of  the  battery  311012 
divided  by  the  number  of  battery 
sets  to  insure  ready  availability. 

For  battery  chemistries  that  exhibit 
memory,  the  recharge  time  will  be 
lengthened  by  a  mandatory 
complete  discharge  prior  to 
commencement  of  charging. _ 


Battery  Leakage  -  Derived 

This  is  the  discharge  rate  under  no  311011 

load  conditions.  The  lower  the  rate 

the  better  -  a  goal  should  be  a  no 

load  discharge  rate  of  1 000  times 

less  than  when  fully  operational.  _ 


20 

While  battery  comes 
in  wide  range  of  form 
factors,  solution 
would  require  a  large 
and  bulky  pack. 

100 

Capable  of  building 
cell  pack  to  deliver 
required  voltage. 

8 

Rated  poorly,  because 
the  pack  will  need  16 
hours  to  charge, 
necessitating  multiple 
packs  per  unit. 

30 

Poor  charge  retention 
in  models 
investigated. 
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Table  A-4-4  Energy  Sources  Trade  Study:  Nickel  Metal-Hydride  (Continued) 


System  Standby  Power 

Derived 

Consumption  (SSPC)  (i.e.  sleep 

311011 

mode)  -  should  be  100  times 

longer  than  operational  time.  E.g. 

if  operational  time  is  2  hours,  then 

SSPC  should  be  roughly  a  week. 

10  70  I  Meets  specification. 


Shelf  Life  -  at  least  1  year 
within  temperature  range  -30C  to 
50C 

Derived 

311011 

5 

3 

15 

Operating  Temperatures  -  at 

minimum  should  operate  with  - 
20C  -  +40C  with  negligible 
performance  degradation 

Derived 

311011 

10 

3 

30 

Transportation  -  Are  there  any 
restrictions  on  modes  of 
transportation? 

Derived 

311022 

6 

10 

60 

Safety  - 

1)  if  battery  is  shorted  will  it  -  a) 
rupture?  b)  explode? 

2)  Does  it  vent  toxic  gases 

3)  Does  it  leak  its  contents  over 
time? 

Derived 

311022 

8 

10 

80 

Meets  specification. 


Considered  dry  cell. 
No  transportation 
restrictions 


No  listed  safety 
hazards. 


3.0  Maintainabili 


Disposability  - 

a)  recyclable 

b)  landfill  safe 


Derived  8 
301201 


No  disposal 
restrictions.  Landfill 
safe. 
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Table  A-4-4  Energy  Sources  Trade  Study:  Nickel  Metal-Hydride  (Continued)  | 

4.0  Producibility 

Logistics  -  Are  the  batteries 
readily  available?  (Are  they  a 
commodity  product  available  from 
several  sources?) 

Derived 

311011 

9 

10 

90 

Available  from 
several  manufacturers. 

Cost  -  For  greatest  economy, 
minimize  the  cost  per  operational 
usage  hour.  I.e.  cost  divided  by  the 
operating  time,  where  maximum 
operating  time  =  (cyclability  * 
(capacity/amps).  Cyclability 
typically  ranges  from  1  (for  a  non- 
rechargeable  battery)  to  1000. 

Derived 

311011 

8 

10 

80 

-$275.00 

Total: 

759 

This  trade  study  is  for  energy  sources  for  the  ITI-ALC  Maintenance  Support  Device  (MSD).  The  goal 
for  the  MSD  is  sufficient  capacity  to  operate  a  complete  8-hour  shift  with  at  most  one  battery  change. 
For  example,  the  system  duty  cycle  for  an  inspection  task  may  range  from  as  low  as  20%  to  as  high  as 
100%  but  will  be  a  function  of  the  logistics  activity  (which  is  TBD).  Similar  variations  are  possible  with 
other  task  types.  The  trade  study  criteria  define  a  decision  procedure  for  choosing  a  battery  once  the 
weight,  dominant  voltage,  and  needed  capacity  of  the  overall  system  is  known.  Hence,  several  criteria 
are  described  by  equations  rather  than  absolute  ranges  of  values.  Once  these  parameters  are  known 
with  greater  precision  (from  the  concurrent  work  on  the  wearable  computer  trade  studies),  we  will  be 
able  to  choose/design  the  battery  pack  with  greater  precision. 
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A-5.  Wireless  Lan 


1.0  Objective 

A  survey  of  the  current  wireless  LAN  technologies  to  be  used  utilized  for  a  specific  application. 
Particularly  for  an  aircraft  hangar  environment  in  which  it  will  be  used  to  transmit  text,  forms, 
graphics,  images,  audio,  and  web  browser  activity.  The  product  must  have  adequate 
bandwidth  to  be  capable  of  performing  all  these  tasks  in  a  reasonable  amount  of  time. 
Evaluation  of  the  performance  and  features  of  these  systems  is  an  important  part  of  the 
survey.  Along  with  that,  the  adaptability  of  the  system  to  the  specific  functional  requirements  is 
the  other  part  that  will  be  surveyed.  One  of  the  desirable  features  of  the  radios  is  a  small  form 
factor.  This  feature  will  allow  the  radio  to  be  discretely  incorporated  with  the  hand  held  unit.  As 
we  will  see,  the  form  factor  will  play  an  important  role  in  the  network  architecture  decision.  The 
current  technologies  that  we  will  be  considering  are  all  spread  spectrum  radios,  three  of  them 
are  frequency  hopping  spread  spectrum  and  one  of  them  is  a  direct  sequence  spread 
spectrum  radio. 

The  four  radios  that  we  will  be  surveying: 

•  Proxim  RangeLAN2  (Table  A-5-2) 

•  Aironet  ARLAN  3000  (Table  A-5-3) 

•  Symbol  Spectrum-24  (Table  A-5-4) 

•  Lucent  WaveLAN  (Table  A-5-5) 

The  final  recommendation  will  be  based  on  how  well  these  products  would  comply  with  the 
requirements  of  the  wireless  system  in  a  specialized  environment.  Therefore,  we  have  a  matrix 
that  outlines  the  most  relevant  characteristics  and  their  level  of  importance  to  the  specific 
environment  and  function.  Each  system  will  then  be  graded  on  each  characteristic.  Along  with 
this  we  will  be  doing  performance  testing.  Through  careful  analysis  of  the  data  from  both  parts 
of  the  survey  we  will  get  a  good  understanding  of  all  the  necessary  elements  of  the  ideal 
system  for  this  specific  function. 


2.0  Procedure 


Product  description  matrices  will  be  filled  out  with  data  that  is  collected  experimentally  and 
specifications  that  were  supplied  by  the  manufacturer.  These  matrices  are  oriented  towards 
the  specific  application  of  the  wireless  LAN.  These  are  some  of  the  elements  in  the  matrix: 

•  Bandwidth 

•  EMI  Interference 
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•  EMI  Immunity 

•  Power  Management 

•  Power  Requirements 

•  Resistance  to  Noise  and  Crosstalk 

•  Error  Detection  and  Correction 

•  Adaptability 

•  Ease  of  Installation/Expandability 

•  Topology  of  Infrastructure 

•  Coverage 

•  Signal  Extendibility 

•  Interoperability 

•  Software  Support 

•  Testability 

•  Diversity  Antenna 

•  Coverage 

•  Antenna  Integration 

•  Radio  Integration 

After  these  matrices  will  be  filled  out,  there  will  be  a  total  score  for  each  system  and  this  score 
will  represent  how  well  the  particular  system  matches  the  requirements  for  the  specific 
application.  Each  system  will  have  its  own  score  and  they  can  be  compared  with  one  another 
to  determine  the  most  suitable  system. 

The  experimental  data  will  be  collected  from  running  tests  in  two  environments:  in  an  office 
environment  with  many  walls  and  small  spaces,  and  in  a  hangar  environment  with  open  space 
but  many  large  metallic  objects  within  the  space.  All  the  experiments  will  be  conducted  with  a 
common  set  of  benchmarks  and  testing  procedures.  Through  these  experiments  we  will  be 
most  interested  in  the  range,  throughput,  and  signal  strength  that  we  will  be  getting  from  the 
system. 

The  benchmarks  that  we  will  be  using  are  files  of  size  339KB,  1.3MB,  3.8MB,  and  5.6MB.  We 
will  be  then  FTP’ing  files  across  this  wireless  network.  The  network  is  composed  of  a  PC, 
which  is  attached  on  a  wired  network,  and  then  to  the  access  point.  The  other  side  of  the 
network  includes  a  laptop  with  the  radio  card,  which  will  be  communicating  with  the  access 
point  and  the  PC  ftp  server.  The  testing  will  proceed  by  using  FTP  at  various  ranges,  and  then 
at  each  range  we  run  the  benchmark  files. 

Ranges  tested: 

5  ft. 

25  ft. 

60  ft. 

-120  ft. 
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File  sizes: 

339KB 

1.3MB 

3.8MB 

5.6MB. 

The  information  in  the  matrix  that  is  not  found  experimentally  was  obtained  from  the  respective 
manufacturers.  Finally  each  score  in  each  category  will  be  multiplied  by  the  corresponding 
weights  and  the  final  total  weighted  sum  will  be  calculated. 

3.0  Analysis 


Here  we  present  our  experimental  results,  which  will  show  us  how  each  system  performed 
based  on  the  matrix  score  and  the  throughput  testing.  Our  analysis  indicates  a  discrepancy 
between  the  system  that  scored  highest  in  the  matrix  and  the  system  that  performed  best  in 
throughput.  This  is  a  very  important  issue  because  someone  can  incorrectly  select  a  final 
product  by  only  looking  at  the  highest  tested  throughput  results  or  just  the  highest  score  based 
on  the  matrix.  The  correct  way  to  choose  will  be  to  look:  at  the  results  that  the  matrices 
produce,  and  the  performance  tests.  The  ideal  system  will  be  chosen  after  a  compromise, 
between  the  matrix  score  and  the  performance  results,  is  reached.  This  compromise  will 
select  the  system  that  will  most  fulfill  the  overall  requirements  of  the  specific  application. 

After  analyzing  the  final  weighted  score  from  the  matrices  we  see  that  Aironet  (score=1215, 
Figure  A-5-1.)  and  Symbol  (score=1186,  Figure  A-5-1.)  are  clear  winners  as  the  overall  best 
products  to  match  the  requirements  of  this  specific  application  (see  below). 


158 


Figure  A-5-1.  Product  Survey  Matrix  Weighted  Scores 


The  performance  tests  show  us  some  dramatic  differences  between  systems.  Looking  at  the 
throughput  results  for  the  office  environment  (Figure  A-5-2.),  we  see  that  Lucent  (1.19Mbps 
avg.,  1.45Mbps  max.,  Table  A-5-1.)  and  Symbol  (604kbps  avg.,  650kbps  max.,  Table  A-5-1.) 
are  the  clear  winners  in  this  category  (see  below).  Looking  at  the  results  for  the  hangar 
environment  (Figure  A-5-3.),  we  see  that  Lucent  (1.16Mbps  avg.,  1.66Mbps  max.  Table  A-5-1.) 
and  Symbol  (628kbps  avg.,  650kbps  max.,  Table  A-5-1.)  are  clear  winners  in  this  category 
(see  below).  (Proxim  hardware  was  not  available  therefore  hangar  environment  testing  was  not 
completed). 
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Figure  A-5-2.  Office  Environment  Testing  Results 


Figure  A-5-3.  Hangar  Environment  Testing  Results 

Some  further  analysis  on  the  hangar  environment  testing  shows  some  patterns  as  compared 
to  the  office  environment  tests.  We  noticed  that  the  throughput  for  the  hangar  environment  for 
the  first  two  ranges  (5ft.  and  20ft.)  was  larger  than  the  office  environment  at  these  ranges.  This 
was  true  for  the  Aironet  and  the  Symbol  systems.  Then  for  the  other  two  ranges  (50ft.  and 
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~1 20ft.),  the  throughput  for  the  hangar  environment  is  lower  than  the  office  environment.  This 
was  also  true  for  the  Aironet  and  the  Symbol  systems.  I  would  extend  the  same  kind  of  results 
to  the  Proxim  system.  The  common  thread  between  all  these  systems  is  that  they  are  all 
frequency  hopping,  and  I  believe  that  multipath  is  causing  these  patterns.  Theoretically  in  an 
open  environment  the  throughput  should  be  better  for  the  more  direct  line  of  sight,  but  this  is 
only  true  for  shorter  ranges  as  concluded  from  our  results.  Then  as  the  range  increases 
multipath  issues  come  into  effect,  and  since  these  are  frequency-hopping  systems,  it  causes 
the  degradation  in  throughput.  In  the  Lucent  system,  which  is  direct  sequenced,  I  did  not  notice 
these  multipath  effects. 


Table  A-5-1  Avg.  and  Max  Throughput  Results  for  Office  and  Hangar  Environments 


Throughput 

Lucent 

Symbol 

Aironet 

Proxim 

Avg.  Office 

1.19  Mbps 

604  kbps 

518  kbps 

321  kbps 

Avg.  Hangar 

1.16  Mbps 

628  kbps 

509  kbps 

- 

Max  Office 

1 .45  Mbps 

650  kbps 

550  kbps 

390  kbps 

Max  Hangar 

1.69  Mbps 

650  kbps 

530  kbps 

- 
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4.0  Product  Survey  Matrices 

4.1  Proxim  RangeLAN2  7201  PC/  7510  AP-2 


Wireless  Communication 

Table  A-5-2  Trade  Study:  Proxim  RangeLAN2  7201  PC/  7510  AP-2 

Criteria 

Rqmts 

Weig 

ht 

Score 

Total 

Comments 

1.0  Performance 

Bandwidth 

This  is  the  bandwidth  of  the  of  the 
wireless  communication  channel. 
Higher  bandwidth  is  better. 

Current  best  is  2.1  Mb/sec  (1.1 
Mb/sec  actual).  Bandwidths  as 
high  as  5.7  Mb  and  30Mb  will  be 
available  soon,  There  is  often  a 
trade  off  between  coverage  and 
bandwidth  (e.g.  CDPD  at  19.2  has 
greater  coverage  but  much  lower 
bandwidth) 

Derived 

311002 

10 

4 

40 

Advertised  bandwidth 
is: 

1.6  Mbps 

Actual  average 
bandwidth  from  tests 
on  file  sizes  from  .3- 
5.6MB,  and  different 
ranges  came  out  to  be: 
350  kbps  (office). 

Max:  390  kbps. 

Much  lower  bandwidth 
results  than  expected. 

EMI  Interference 

The  wireless  LAN  should  not 
interfere  with  existing  RF 
equipment  in  the  hanger.  Typical 
ranges  for  wireless  is  9600  MHz- 
2.4  GHz.  Should  not  have  a 
smaller  range 

Derived 

311002 

10 

5 

50 

We  had  another 
wireless  system 
present  and  there  was 
no  noticeable  sign  that 
the  Proxim  was 
interfering  with  it. 
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Table  A-5-2  Trade  Study:  Proxim  RangeLAN2  7201  PC/ 7510  AP-2  (Continued) 


EMI  immunity 

The  wireless  LAN  should  be 
minimally  affected  by  existing  RF 
environment 


Derived 

311002 


Power  management 

Wireless  LAN  should  be  capable 
of  adopting  power  management 
techniques  -  sleep  mode,  standby 
mode,  wait  mode  (i.e.  is  there  an 
active  signal),  polling  mode 

Derived 

311002 

10 

7 

70 

Power  requirements 

(on  transmitting  and  receiving 
ends)  -  typical  values  are  1 .7  watts 
receive,  3.4  W  transmit.  Power 
management  should  reduce  this 
requirement.  Lower  values  are 
better. 

Derived 

311002 

9 

6 

54 

2.0  Reliability 

Resistance  to  noise  and  crosstalk 

Does  system  use  direct-sequence 
spread  spectrum  (DSSS)  at  915 
Mhz  or  frequency  hopping  spread 
spectrum  (FHSS)  at  2.4  GHz  or 

5.7  GHz? 

Derived 

311002 

9 

7 

63 

The  lab  we  were 
testing  in  had  several 
access  points  that  were 
also  frequency¬ 
hopping  architectures, 
along  with  a  campus 
wide  wireless  LAN 
that  is  direct 
sequenced.  We  did  not 
see  any  great 
interference  that 
slowed  down  the 
throughput. 


Has  a  power 
management  system 
called:  Marathon 
Power  Management.  It 
has  a  doze  and  a  sleep 
mode. 


It  draws  300  mA  for 
transmit  and  150  mA 
for  receive.  A 
reasonable  value  which 
is  better  than  most 
other  systems. 


This  system  is  a 
frequency  hopping 
spread  spectrum 
(FHSS)  at  2.4GHz. 

The  resistance  to  noise 
and  crosstalk  is  very 
good.  No  noticeable 
signs  in  testing. 
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Table  A-5-2  Trade  Study:  Proxim  RangeLAN2  7201  PC/  7510  AP-2  (Continued) 


Error  detection  and  correction 

What  is  the  number  of  errors  that 
can  be  corrected?  Does  the  system 
employ  methods  such  as 
checksum,  Cycle  Redundancy 
Coding  (CRC),  and  automatic 
recovery  procedures? 

Derived 

311002 

9 

5 

45 

This  was  hard  to  judge 
because  there  is  no 
documentation  on  this 
but  in  our  testing 
things  we  never 
encountered  any 
corrupted  data. 

Adaptability 

The  wireless  network  will  be 
adequately  robust  to  different 
signal  and  environmental  changes. 
It  should  comply  with 
requirements  of  appropriate 
regulatory  authority,  including 

FCC  Part  15.247  and  15.249. 

Derived 

311002 

8 

5 

40 

It  complies  with  FCC 
Part  15  in  the  US, 

ETSI ETS  300.328  and 
CE  EMC-EEC  in 
Europe. 

3.0  Maintainabili 


Ease  of 

Installation/Expandability 

Addition  of  base  station  adapter 
cards  and  network  gateway  units 
should  only  require  configuration 
of  those  units 


Derived  7 
311002 


Topology  of  infrastructure  Derived  9 

How  easily  does  infrastructure  311 002 

support  mobile  communication? 
cellular  (10),  or  dedicated  (6) 


Addition  of  an  access 
point  only  involves 
simple  configuration, 
which  basically  sets 
the  IP  address,  and  as 
for  the  radio,  the 
drivers  need  to  be 
loaded.  Then  the 
system  will  look  for  its 
radios  and  start 
communicating 


With  a  single  access 
point  there  is  a  limit  to 
the  range  of  operability 
but  with  multiple 
access  points,  roaming 
will  allow  you  to  move 
freely  almost 
anywhere. 
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Table  A-5-2  Trade  Study:  Proxim  RangeLAN2  7201 
Coverage  Derived  7  8 

Do  gateway  units  support  3 1 1 002 

diversity  antennae,  directional 
antennae  etc.?  How  easily  is 
coverage  expanded  through  use  of 
improved  antennae  technology 

Signal  Extendibility  Derived  8  8 

Can  infrastructure  be  extended  3 1 1 002 

such  that  signals  of  low  coverage 

can  be  brought  into  the  wireless 

network? 


4.0  Producability 


PC/  7510  AP-2  (Continued) 

56  No  diversity  antennae. 

Basically  switching 
antennas  and  some 
minimal  configuration 
on  the  access  point  will 
allow  for  greater 

_ coverage. _ 

64  This  is  easily  done 

with  the  addition  of  an 
access  point  at  the 
appropriate  location  to 
pick  up  those  specific 
signals. 


Interoperability 

Is  there  IEEE  802. 1 1  Wireless 

LAN  support? 

Derived 

311002 

9 

5 

45 

There  is 

interoperability  with 
Wireless  LAN 
Interoperability  Forum 
(WLI  Forum)  products. 
There  is  no  specific 
mention  of  IEEE 

802.1 1  support.  In 
testing  we  were  not 
able  to  get  it  to 
interoperate. 

Software  Support 

Are  drivers,  configuration 
software,  and  benchmark  software 
available  for  all  targeted 
platforms? 

Derived 

311002 

10 

7 

70 

Good  support  of  most 
of  the  major  platforms: 
Win95,  NT,  DOS.  No 
support  for  WinCE, 
Newton,  Unix. 
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Table  A-5-2  Trade  Study:  Proxim  RangeLAN2  7201 


Testability  Derived  9  10 

The  wireless  system  should  come  3 1 1 002 

with  adequate  benchmark 

software  to  ascertain  quality  of 

signal  (strength  of  signal,  signal 

degradation,  coverage) 


Diversity  Antennae 

Do  radios  support  diversity 
antennae? 


Coverage 

LAN  needs  to  cover  people 
working  around  plane  to  a  local 
server.  Currently  best  wireless 
LAN  spread  spectrum  delivers  a 
600  ft  radius  outside,  300  ft  inside 
dependent  on  layout.  Repeaters 
can  typically  be  used  to  extend 
signal. 


Derived 

311002 


Derived  8 
311002 


Antenna  Integration 
How  easily  can  antenna  be 
integrated  with  wearable 
platform? 


Derived  10 
311002 


90 

Very  good 

benchmarking  software 
that  made  system  and 
site  surveys  really 
clear.  Good  GUI’s  for 
other  signal 
interference,  signal 
strength,  and 
throughput  analysis. 

0 

There  is  no  support  for 
diversity  antenna. 

32 

Advertised  coverage 
with  the  tested  snap-on 
antenna:  700  ft.  (open) 
and  400  ft.  (office). 

Test  results  with  snap- 
on  antenna  showed 
coverage  of:  300  ft. 
(open)  and  80  ft. 

(office). 

Our  office  setting  had 
many  walls  and  doors. 

90 

The  antenna  fits  very 
nicely.  It  is  just  an 
extension  of  the  radio’s 
PCMCIA  card.  It  is  the 
width  and  height  of  PC 
card  and  maybe  1” 
extension  beyond  PC 
card  edge. 
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Table  A-5-2  Trade  Study:  Proxim  RangeLAN2  7201  PC/  751' 

AP-2  (Continued) 

Radio  Integration 

Will  radio  fit  within  wearable 
platform? 

Derived 

311002 

10 

9 

90 

The  radio  is  a 

PCMCIA  card,  which 
is  ideal  for  a  wearable 
platform  assuming  it 
has  a  PCMCIA  slot. 

The  antenna  has  a  very 
small  form  factor. 
Although  it  does 
protrude  out  of  the 
radio  edge,  it  can  be 
incorporated  into  the 
package  very  easily. 

Total: 

123 

1099 
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4.2  Aironet  ARLAN  3000  PC3000/ AP3000-E 


Wireless  Communications 

Table  A-5-3  Trade  Study:  Aironet  ARLAN  3000  PC3000/  AP3000-E 

Criteria 

Rqmts 

Weight 

Score 

Total 

Comments 

1.0  Performance 

Bandwidth 

This  is  the  bandwidth  of  the  of  the 
wireless  communication  channel. 
Higher  bandwidth  is  better. 

Current  best  is  2.1  Mb/sec  (1.1 
Mb/sec  actual).  Bandwidths  as 
high  as  5.7  Mb  and  30Mb  will  be 
available  soon.  There  is  often  a 
trade  off  between  coverage  and 
bandwidth  (e.g.  CDPD  at  19.2  has 
greater  coverage  but  much  lower 
bandwidth) 

Derived 

311002 

10 

6 

60 

Advertised  bandwidth 
is: 

1 .0  Mbps 

Actual  average 
bandwidth  from  tests 
on  file  sizes  from  .3- 
5.6MB,  and  different 
ranges  came  out  to 
be:  518kbps  (office), 
509  kbps  (hangar). 
Max:  550  kbps 
(office);  530  kbps 
(hangar). 

This  bandwidth  is 
better  than  expected 
and  50%  of  what  is 
advertised  is  real 
good. 

EMI  Interference 

The  wireless  LAN  should  not 
interfere  with  existing  RF 
equipment  in  the  hanger.  Typical 
ranges  for  wireless  is  9600  MHz- 
2.4  GHz.  Should  not  have  a 
smaller  range 

Derived 

311002 

10 

5 

50 

We  had  another 
wireless  system 
present  and  there  was 
no  noticeable  sign 
that  the  Aironet  was 
interfering  with  it. 
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Table  A-5-3  Trade  Study:  Aironet  ARLAN  3000  PC3000/  AP3000-E  (Continued) 


EMI  immunity 

The  wireless  LAN  should  be 
minimally  affected  by  existing  RF 
environment. 


Derived 

311002 


Power  management 

Wireless  LAN  should  be  capable 
of  adopting  power  management 
techniques  -  sleep  mode,  standby 
mode,  wait  mode  (i.e.  is  there  an 
active  signal),  polling  mode. 


Derived  1 0 
311002 


The  lab  we  were 
testing  in  had  several 
access  points  that 
were  also  frequency¬ 
hopping  architectures, 
along  with  a  campus 
wide  wireless  LAN 
that  is  direct 
sequenced.  We  did 
not  see  any  great 
interference  that 
slowed  down  the 

throughput. _ 

It  has  a  basic  power 
management  system. 
There  is  no  mention 
of  a  sleep  or  doze 
mode.  Also  we 
noticed  that  our 
laptops  battery 


Power  requirements 

Derived 

9 

6 

54 

It  draws  210mA  on 

(on  transmitting  and  receiving 

311002 

receive,  and  450  mA 

ends)  -  typical  values  are  1 .7  watts 

on  transmit.  These 

receive,  3.4  W  transmit.  Power 

numbers  are  in  the 

management  should  reduce  this 

mid-range  of  the 

requirement.  Lower  values  are 

systems  tested,  but 

better. 

are  still  good  overall. 

2.0  Reliability _ 

Resistance  to  noise  and  crosstalk  Derived 
Does  system  use  direct-sequence  3 1 1 002 
spread  spectrum  (DSSS)  at  915 
Mhz  or  frequency  hopping  spread 
spectrum  (FHSS)  at  2.4  GHz  or 
5.7  GHz? 


This  system  is  a 
frequency  hopping 
spread  spectrum 
(FHSS)  at  2.4GHz. 
The  resistance  to 
noise  and  crosstalk  is 
very  good.  No 
noticeable  signs  in 
testing. _ 
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Table  A-5-3  Trade  Study:  Aironet  ARLAN 


Error  detection  and  correction 

What  is  the  number  of  errors  that 
can  be  corrected?  Does  the  system 
employ  methods  such  as 
checksum,  Cycle  Redundancy 
Coding  (CRC),  and  automatic 
recovery  procedures? 


Adaptability 

The  wireless  network  will  be 
adequately  robust  to  different 
signal  and  environmental  changes. 
It  should  comply  with 
requirements  of  appropriate 
regulatory  authority,  including 
FCC  Part  15.247  and  15.249. 


3.0  Maintainabili 


Ease  of 

Installation/Expandability 

Addition  of  base  station  adapter 
cards  and  network  gateway  units 
should  only  require  configuration 
of  those  units. 


Derived 

311002 


PC3000/  AP3000-E  (Continued) 


45  This  was  hard  to 

judge  because  there  is 
no  documentation  on 
this  but  in  our  testing 
things  we  never 
encountered  any 


8 

64 

It  complies  with  FCC 
Part  15,  Subpart  B, 
Class  B  and  FCC  Part 

15.247. 

Derived  7 
311002 


Topology  of  infrastructure 

How  easily  does  infrastructure 
support  mobile  communication? 
cellular  (10),  or  dedicated  (6) 


Derived 

311002 


Addition  of  an  access 
point  only  involves 
simple  configuration, 
which  basically  sets 
the  IP  address,  and  as 
for  the  radio,  the 
drivers  need  to  be 
loaded.  Then  the 
system  will  look  for 
its  radios  and  start 
communicating _ 


With  a  single  access 
point  there  is  a  limit 
to  the  range  of 
operability  but  with 
multiple  access 
points,  roaming  will 
allow  you  to  move 
freely  almost 
anywhere. _ 


1  Table  A-5-3  Trade  Study:  Aironet  ARLAN  300C 

PC3000/  AP3000-E  (Continued) 

Coverage 

Do  gateway  units  support 
diversity  antennae,  directional 
antennae  etc.?  How  easily  is 
coverage  expanded  through  use  of 
improved  antemiae  technology? 

Derived 

311002 

7 

10 

70 

Supports  diversity, 
omni-directional,  and 
directional.  Basically 
switching  antennas 
and  some  minimal 
configuration  on  the 
access  point  will 
allow  for  greater 
coverage. 

Signal  Extendibility 

Can  infrastructure  be  extended 
such  that  signals  of  low  coverage 
can  be  brought  into  the  wireless 
network? 

Derived 

311002 

8 

8 

64 

This  is  easily  done 
with  the  addition  of 
an  access  point  at  the 
appropriate  location 
to  pick  up  those 
specific  signals. 

4.0  Producability 

NA 

5.0  Capability 

Interoperability 

Is  there  IEEE  802.1 1  Wireless 

LAN  support? 

Derived 

311002 

9 

6 

54 

The  ARLAN  3000 
series  products  were 
based  on  draft  D5  of 
the  IEEE  802.11 
specification.  Aironet 
says  that  they  will 
comply  with  this 
standard. 

Software  Support 

Derived 

10 

8 

80 

Good  support  of  most 

Are  drivers,  configuration 

311002 

of  the  major 

software,  and  benchmark  software 

platforms:  Win95, 

available  for  all  targeted 

NT,  DOS.  OS/2.  No 

platforms? 

support  for  WinCE, 
Newton. 
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Table  A-5-3  Trade  Study;  Aironet  ARLAN  3000  PC3000/  AP3000-E  (Continued) 


Testability 

The  wireless  system  should  come 
with  adequate  benchmark 
software  to  ascertain  quality  of 
signal  (strength  of  signal,  signal 
degradation,  coverage) 

Derived 

311002 

9 

8 

72 

Basic  benchmarking 
software  that  can  be 
accessed  through  the 
access  point. 

Diversity  Antennae 

Do  radios  support  diversity 
antennae? 

Derived 

311002 

6 

10 

60 

Supports  a  diversity 
antenna  that  connects 
externally  to  the 
radio. 

Coverage 

LAN  needs  to  cover  people 
working  around  plane  to  a  local 
server.  Currently  best  wireless 
LAN  spread  spectrum  delivers  a 
600  ft  radius  outside,  300  ft  inside 
dependent  on  layout.  Repeaters 
can  typically  be  used  to  extend 
signal 

Derived 

311002 

8 

6 

48 

Advertised  coverage 
with  the  tested  snap- 
on  antenna:  1 000  ft. 
(open)  and  500  ft. 
(office). 

Test  results  with 
snap-on  antenna 
showed  coverage  of: 
500  ft.  (open)  and  130 
ft.  (office). 

Our  office  setting  had 
many  walls  and 
doors. 

Antenna  Integration 

How  easily  can  antenna  be 
integrated  with  wearable 
platform? 

Derived 

311002 

10 

9 

90 

The  antenna  fits  very 
nicely.  It  is  just  an 
extension  of  the 
radio’s  PCMCIA 
card.  It  is  the  width 
and  height  of  PC  card 
and  maybe  1” 
extension  beyond  PC 
card  edge. 
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Table  A-5-3  Trade  Study:  Aironet  ARLAN  300C 

PC300I 

9/  AP3( 

100-E  (Continued) 

Radio  Integration 

Will  radio  fit  within  wearable 
platform? 

Derived 

311002 

10 

9 

90 

The  radio  is  a 

PCMCIA  card,  which 
is  ideal  for  a  wearable 
platform  assuming  it 
has  a  PCMCIA  slot. 
The  antenna  has  a 
very  small  form 
factor.  Although  it 
does  protrude  out  of 
the  radio  edge,  it  can 
be  incorporated  into 
the  package  very 
easily. 

Total: 

140 

1215 

I 
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4.3  Symbol  Spectrum-24  LA  2400/  AP  2410 


Wireless  Communication 

Table  A-5-4  Trade  Study:  Symbol  Spectrum-24  LA  2400/  AP  2410 

Criteria 

Score 

Total 

Comments 

1.0  Performance 

Bandwidth 

This  is  the  bandwidth  of  the  of  the 
wireless  communication  channel. 
Higher  bandwidth  is  better. 

Current  best  is  2.1  Mb/sec  (1.1 
Mb/sec  actual).  Bandwidths  as 
high  as  5.7  Mb  and  30Mb  will  be 
available  soon,  There  is  often  a 
trade  off  between  coverage  and 
bandwidth  (e.g.  CDPD  at  19.2  has 
greater  coverage  but  much  lower 
bandwidth) 

Derived 

311002 

10 

7 

70 

Advertised  bandwidth 
is: 

1.0  Mbps 

Actual  average 
bandwidth  from  tests 
on  file  sizes  from  .3- 
5.6MB,  and  different 
ranges  came  out  to 
be:  604  kbps  (office), 
628  kbps  (hangar). 

Max:  650  kbps  (office 
&  hangar). 

Even  though  it  did 
not  reach  its  raw  data 
throughput  it  did  have 
an  impressive 
throughput  that  is 
>50%  of  what  they 
advertise.  Did  much 
better  than  expected. 

EMI  Interference 

The  wireless  LAN  should  not 
interfere  with  existing  RF 
equipment  in  the  hanger.  Typical 
ranges  for  wireless  is  9600  MHz- 
2.4  GHz.  Should  not  have  a 
smaller  range 

Derived 

311002 

10 

6 

We  had  another 
wireless  system 
present  and  there  was 
no  noticeable  sign 
that  the  Symbol  was 
interfering  with  it. 
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Tabic  A-5-4  Trade  Study:  Symbol  Spectrum- 


EMI  immunity  Derived  9 

The  wireless  LAN  should  be  311 002 

minimally  affected  by  existing  RF 
environment 


Derived  10 
311002 


Power  management 

Wireless  LAN  should  be  capable 
of  adopting  power  management 
techniques  -  sleep  mode,  standby 
mode,  wait  mode  (i.e.  is  there  an 
active  signal),  polling  mode 


Power  requirements  Derived 

(on  transmitting  and  receiving  3 1 1 002 

ends)  -  typical  values  are  1 .7  watts 
receive,  3.4  W  transmit.  Power 
management  should  reduce  this 
requirement.  Lower  values  are 
better. 


24  LA  2400/  AP  2410  (Continued) 


6  54  The  lab  we  were 

testing  in  had  several 
access  points  that 
were  also  frequency¬ 
hopping  architectures, 
along  with  a  campus 
wide  wireless  LAN 
that  is  direct 
sequenced.  We  did 
not  see  any  great 
interference  that 
slowed  down  the 

_ throughput. _ 

7  70  It  does  have  a  power 

saving  mode  that 
significantly  reduces 
the  current  draw  for 
both  transmit  and 
receive.  Also  in  the 
continuous  aware 
mode  it  does  have  a 
sleep  mode. 


5  45  For  receive:  100m W: 

300mA. 

For  transmit: 
lOOmW:  400mA 

These  numbers  are 
higher  than  the  other 
systems.  Still  they  are 
good  with  relation  to 
the  overall  typical 
values. 
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Table  A-5-4  Trade  Study:  Symbol  Spectrum-24  LA  2400/  AP  2410  (Continued) 


Resistance  to  noise  and  crosstalk 

Does  system  use  direct-sequence 
spread  spectrum  (DSSS)  at  915 
Mhz  or  frequency  hopping  spread 
spectrum  (FUSS)  at  2.4  GHz  or 

5.7  GHz? 

Derived 

311002 

9 

7 

63 

This  system  is  a 
frequency  hopping 
spread  spectrum 
(FHSS)  at  2.4GHz. 
The  resistance  to 
noise  and  crosstalk  is 
very  good.  No 
noticeable  signs  in 
testing. 

Error  detection  and  correction 

What  is  the  number  of  errors  that 
can  be  corrected?  Does  the  system 
employ  methods  such  as 
checksum,  Cycle  Redundancy 
Coding  (CRC),  and  automatic 
recovery  procedures? 

Derived 

311002 

9 

5 

45 

This  was  hard  to 
judge  because  there  is 
no  documentation  on 
this  but  in  our  testing 
things  we  never 
encountered  any 
corrupted  data. 

Adaptability 

The  wireless  network  will  be 
adequately  robust  to  different 
signal  and  environmental  changes. 
It  should  comply  with 
requirements  of  appropriate 
regulatory  authority,  including 

FCC  Part  15.247  and  15.249. 

Derived 

311002 

8 

8 

64 

It  complies  with  FCC 
Part  15.247,  15.205, 
15.209  in  the  US, 

ETS  300.328  in 
Europe. 

3.0  Maintainabili 


Ease  of 

Installation/Expandability 

Addition  of  base  station  adapter 
cards  and  network  gateway  units 
should  only  require  configuration 
of  those  units 


Derived  7 
311002 


Addition  of  an  access 
point  only  involves 
simple  configuration, 
which  basically  sets 
the  IP  address,  and  as 
for  the  radio,  the 
drivers  need  to  be 
loaded.  Then  the 
system  will  look  for 
its  radios  and  start 
communicating 
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Table  A-5-4  Trade  Study:  Symbol  Spectrum 
Topology  of  infrastructure  Derived  9 

How  easily  does  infrastructure  3 1 1 002 
support  mobile  communication? 
cellular  (10),  or  dedicated  (6) 


Derived  7 
311002 


Coverage 

Do  gateway  units  support 
diversity  antennae,  directional 
antennae  etc.?  How  easily  is 
coverage  expanded  through  use  of 
improved  antennae  technology 


Signal  Extendibility  Derived 

Can  infrastructure  be  extended  311 002 

such  that  signals  of  low  coverage 
can  be  brought  into  the  wireless 
network? 


24  LA  2400/  AP  2410  (Continued) 

7  63  With  a  single  access 

point  there  is  a  limit 
to  the  range  of 
operability  but  with 
multiple  access 
points,  roaming  will 
allow  you  to  move 
freely  almost 
anywhere.  The 
negative  point  here  is 
that  it  does  not 
support  peer-to-peer 
communication 
without  a  micro 
_ access  point. _ 


70  Directional  and 
diversity  support. 
Basically  switching 
antennas  and  some 
minimal 

configuration  on  the 
access  point  will 
allow  for  greater 
coverage. 


64  This  is  easily  done 
with  the  addition  of 
an  access  point  at  the 
appropriate  location 
to  pick  up  those 
specific  signals.  This 
can  also  be  done  with 
the  addition  of  micro 
access  points. 


4.0  Producability 
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Table  A-5-4  Trade  Study:  Symbol  Spectrum-24  LA  2400/  AP  2410  (Continued 


MiftfWIfflfffS 


Interoperability 

Is  there  IEEE  802.1 1  Wireless 
LAN  support? 

Derived 

311002 

9 

6 

54 

Symbol  announced 
that  they  would 
incorporate  the 

802.1 1  standard  into 
its  Spectrum24-based 
wireless  network 
product  line.  This 
system  is  part  of  the 
Spectrum-24  product 
line. 

Software  Support 

Are  drivers,  configuration 
software,  and  benchmark  software 
available  for  all  targeted 
platforms? 

Derived 

311002 

10 

6 

60 

Good  support  of  most 
of  the  major 
platforms:  Win95, 

NT,  DOS.  No  support 
for  WinCE. 

Testability 

The  wireless  system  should  come 
with  adequate  benchmark 
software  to  ascertain  quality  of 
signal  (strength  of  signal,  signal 
degradation,  coverage) 

Derived 

311002 

9 

8 

72 

Adequate 

benchmarking  tools 
that  can  only  be 
accessed  through  the 
access  point  and  a 
serial  cable.  The 
software  that  is  there 
tells  you  all  the 
important  things  that 
you  want  to  know. 

Diversity  Antennae 

Do  radios  support  diversity 
antennae? 

Derived 

311002 

6 

10 

60 

Support  for  diversity 
antenna. 
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Table  A-5-4  Trade  Study:  Symbol  Spectrum-24  LA  2400/  AP  2410  (Continued) 

Radio  Integration 

Will  radio  fit  within  wearable 
platform? 

Derived 

311002 

10 

8 

80 

The  radio  is  a 
PCMCIA  card,  which 
is  ideal  for  a  wearable 
platform  assuming  it 
has  a  PCMCIA  slot. 
The  antenna  is  the 
only  thing  that  might 
be  harder  to  fit  into 
the  package  because 
it  is  quite  bulky.  It 
seems  to  protrude  too 
much  from  the  edge 
of  the  card  and  the 
card  host. 

Total: 

137 

1186 
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4.4  Lucent  WaveLAN/  WavePOINT 


Wireless  Communication 

Table  A-5-5  Trade  Study:  Lucent  WaveLAN/WavePOINT 


Criteria 

Rqmts 

Weig 

ht 

Score 

Total 

Comments 

1.0  Performance 

Bandwidth 

This  is  the  bandwidth  of  the  of  the 
wireless  communication  channel. 
Higher  bandwidth  is  better. 

Current  best  is  2.1  Mb/sec  (1.1 
Mb/sec  actual).  Bandwidths  as 
high  as  5.7  Mb  and  30Mb  will  be 
available  soon,  There  is  often  a 
trade  off  between  coverage  and 
bandwidth  (e.g.  CDPD  at  19.2  has 
greater  coverage  but  much  lower 
bandwidth) 

Derived 

311002 

10 

1 

10 

100 

Advertised  bandwidth 
is: 

2.0  Mbps 

Actual  average 
bandwidth  from  tests 
on  file  sizes  from  .3- 
5.6MB,  and  different 
ranges  came  out  to  be: 
1.19  Mbps  (office), 

1.16  Mbps  (hangar). 
Max:  1.45  Mbps 
(office);  1 .66  Mbps 
(hangar). 

Throughputs  of  1 

Mbps  are  almost  a 
minimum  and  at  some 
points  it  was  1 .66 

Mbps. 

These  are  very  good 
results  overall. 

EMI  Interference 

The  wireless  LAN  should  not 
interfere  with  existing  RF 
equipment  in  the  hanger.  Typical 
ranges  for  wireless  is  9600  MHz- 
2.4  GHz.  Should  not  have  a 
smaller  range 

Derived 

311002 

10 

9 

90 

We  had  another 
wireless  system 
present  and  there  was 
no  noticeable  sign  that 
the  WaveLAN  was 
interfering  with  it. 
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Table  A-5-5  Trade  Study:  Lucent  WaveLAN/WavePOINT  (Continued) 


EMI  immunity 

The  wireless  LAN  should  be 
minimally  affected  by  existing  RF 
environment 

Derived 

311002 

9 

9 

81 

The  lab  we  were 
testing  in  had  several 
access  points  that  were 
frequency-hopping 
architectures.  We  did 
not  notice  interference, 
also  in  the  hangar 
environment  the  direct 
sequence  was  least 
effected  while  others 
affected  by  multi-path. 

Power  management 

Wireless  LAN  should  be  capable 
of  adopting  power  management 
techniques  -  sleep  mode,  standby 
mode,  wait  mode  (i.e.  is  there  an 
active  signal),  polling  mode 

Derived 

311002 

10 

6 

60 

It  has  a  basic  power 
management  system. 
There  is  mention  of  a 
sleep  mode. 

Power  requirements 

(on  transmitting  and  receiving 
ends)  -  typical  values  are  1.7  watts 
receive,  3.4  W  transmit.  Power 
management  should  reduce  this 
requirement.  Lower  values  are 
better. 

Derived 

311002 

9 

5 

45 

It  draws  296mA  on 
receive,  and  600  mA 
on  transmit.  These 
numbers  are  in  the 
high-range  of  the 
systems  tested,  but  are 
still  good  overall. 

2.0  Reliabilii 


Resistance  to  noise  and  crosstalk 

Derived 

9 

5 

45 

This  system  is  a  direct 

Does  system  use  direct-sequence 

311002 

sequence  spread 

spread  spectrum  (DSSS)  at  915 

spectrum  (DSSS)  at 

Mhz  or  frequency  hopping  spread 

915MHz.  We  saw 

spectrum  (FHSS)  at  2.4  GHz  or 

5.7  GHz? 

some  interference. 
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Table  A-5-5  Trade  Study:  Lucent  WaveLAN/WavePOINT  (Continued) 


Error  detection  and  correction 

Derived 

9 

5 

45 

This  was  hard  to  judge 

What  is  the  number  of  errors  that 

311002 

because  there  is  no 

can  be  corrected?  Does  the  system 

documentation  on  this 

employ  methods  such  as 

but  in  our  testing 

checksum,  Cycle  Redundancy 

things  we  never 

Coding  (CRC),  and  automatic 

encountered  any 

recovery  procedures? 

corrupted  data. 

Adaptability 

Derived 

8 

8 

64 

It  complies  with  FCC 

The  wireless  network  will  be 
adequately  robust  to  different 
signal  and  environmental  changes. 
It  should  comply  with 
requirements  of  appropriate 
regulatory  authority,  including 

FCC  Part  15.247  and  15.249. 

311002 

Part  15. 

Ease  of 

Installation/Expandability 

Addition  of  base  station  adapter 
cards  and  network  gateway  units 
should  only  require  configuration 
of  those  units. 

Derived 

311002 

7 

Topology  of  infrastructure 

Derived 

9 

How  easily  does  infrastructure 

311002 

support  mobile  communication? 

cellular  (10),  or  dedicated  (6) 

Addition  of  an  access 
point  only  involves 
simple  configuration, 
which  basically  sets 
the  IP  address,  and  as 
for  the  radio,  the 
drivers  need  to  be 
loaded.  Then  the 
system  will  look  for  its 
radios  and  start 
communicating 


With  a  single  access 
point  there  is  a  limit  to 
the  range  of  operability 
but  with  multiple 
access  points,  roaming 
will  allow  you  to  move 
freely  almost 
anywhere. 


Table  A-5-5  Trade  Study:  Lucent  WaveLAN/W avePOINT  (Continued) 

Coverage 

Do  gateway  units  support 
diversity  antennae,  directional 
antennae  etc.?  How  easily  is 
coverage  expanded  through  use  of 
improved  antennae  technology 

Derived 

311002 

7 

8 

56 

Omni-directional  and 
directional.  Basically 
switching  antennas  and 
some  minimal 
configuration  on  the 
access  point  will  allow 
for  greater  coverage. 

Signal  Extendibility 

Can  infrastructure  be  extended 
such  that  signals  of  low  coverage 
can  be  brought  into  the  wireless 
network? 

Derived 

311002 

8 

8 

64 

This  is  easily  done 
with  the  addition  of  an 
access  point  at  the 
appropriate  location  to 
pick  up  those  specific 
signals. 

4.0  Producability 


Interoperability 

Is  there  IEEE  802. 1 1  Wireless 

LAN  support? 

Derived 

311002 

9 

0 

0 

This  product,  which  is 
an  older  generation 
WaveLAN,  does  not 
support  this  standard; 
there  is  mention  that 
new  generations  will 
comply. 

Software  Support 

Are  drivers,  configuration 
software,  and  benchmark  software 
available  for  all  targeted 
platforms? 

Derived 

311002 

10 

7 

70 

Good  support  of  most 
of  the  major  platforms: 
Win95,  NT,  DOS.No 
support  for  WinCE, 
Newton. 

Testability 

The  wireless  system  should  come 
with  adequate  benchmark 
software  to  ascertain  quality  of 
signal  (strength  of  signal,  signal 
degradation,  coverage) 

Derived 

311002 

9 

8 

72 

Basic  benchmarking 
software.  Package 
called 

WaveMONITOR. 
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Table  A-5-5  Trade  Study: 


Diversity  Antennae 

Do  radios  support  diversity 
antennae? 


Coverage 

LAN  needs  to  cover  people 
working  around  plane  to  a  local 
server.  Currently  best  wireless 
LAN  spread  spectrum  delivers  a 
600  ft  radius  outside,  300  ft  inside 
dependent  on  layout.  Repeaters 
can  typically  be  used  to  extend 
signal 


Lucent  WaveLAN/WavePOINT  (Continued 


Derived  6  0  0  Does  not  support  a 

3 1 1 002  diversity  antenna. 


Derived  8 
311002 


Antenna  Integration 

How  easily  can  antenna  be 
integrated  with  wearable 
platform? 


Derived  10 
311002 


Advertised  coverage 
with  the  tested 
antenna:  800  ft.  (open) 
and  100  ft.  (office). 

Test  results  with  tested 
antenna  showed 
coverage  of:  400  ft. 
(open)  and  90  ft. 
(office). 

Our  office  setting  had 
many  walls  and  doors. 


It  is  possible  to 
integrate  the  antenna 
but  it  is  a  large  form 
factor.  It  has  a 
PCMCIA  card  but  then 
there  is  a  cable  going 
from  the  card  to  a  large 
3”x5”  unit  which  is  the 
antenna.  It  might  be 
very  hard  to 
incorporate  this  piece 
especially  if  the  hand¬ 
held  platform  is  small. 
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Table  A-5-5  Trade  Study:  Lucent  WaveLAN/WavePOINT  (Continued) 

Radio  Integration 

Will  radio  fit  within  wearable 
platform? 

Derived 

311002 

10 

7 

70 

The  radio  is  a 

PCMCIA  card,  which 
is  ideal  for  a  wearable 
platform.  The  card 
becomes  invisible  and 
is  no  problem  with 
integration  just  as  long 
as  there  is  a  PCMCIA 
slot.  The  cable 
connecting  the  radio  is 
quite  bulky. 

Total: 

5.0  Conclusion 

Our  experimental  results  and  the  final  weighted  scores  need  to  be  taken  into  consideration  in 
making  a  final  decision.  Even  though  Lucent  got  the  highest  throughput  (1.16  Mbps  avg.,  Table 
A-5-1.)  and  Aironet  got  the  highest  score  on  the  matrix  (score=1215,  Figure  A-5-1.),  they  both 
did  not  as  well  in  the  other  criteria.  Symbol  was  the  product  that  had  the  second  highest  score 
from  the  matrix  (score=1186,  Figure  A-5-1.)  and  the  second  highest  throughput  (628  kbps 
avg.,  Table  A-5-1).  We  would  recommend  the  Symbol  system  as  our  number  one  choice 
because  of  its  best  overall  performance  and  best  matches  the  requirements  of  the  specific 
implementation.  The  Symbol  unit  does  well  in  the  performance  section  and  it  also  did  well  in 
having  a  good  majority  of  the  features  that  are  required  by  this  application.  Our  second  choice 
would  be  the  Aironet  system  with  the  highest  score  in  the  matrix  and  it  was  third  in  the 
throughput  category.  The  most  convenient  system  with  respect  to  the  integration  to  the 
wearable  platform  would  be  the  Proxim.  Since  the  Fujitsu  Point  510  has  a  built  in  Proxim 
RangeLAN  2  radio,  it  has  the  highest  degree  of  integration  with  the  wearable  platform.  It  might 
not  be  the  leader  in  throughput  but  the  convenience  being  build  into  a  commercial  product  and 
reduced  maintenance  of  the  radio  makes  the  Proxim  a  very  favorable  option. 

It  might  not  be  clear  why  Symbol  was  chosen  ahead  of  Aironet.  If  you  take  the  placement  of 
each  product  in  the  two  categories  (matrix  score  and  throughput)  they  come  out  to  a  tie.  The 
factor  that  made  us  choose  Symbol  was  the  increased  throughput,  because  our  specific 
functioning  of  the  system  might  be  required  to  do  digital  image  transfer  and  this  is  where 
higher  throughput  will  help. 
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A-6.  MIDDLEWARE  TRADE  STUDY 


1.0  Summation 


1.1  Purpose 

The  purpose  of  this  trade  study  is  to  recommend  a  distributed  infrastructure  for  the  ITI-ALC 
demonstration  environment  that  enables  portions  of  the  demonstration  to  talk  to  one  another 
and  to  legacy  systems.  A  robust  environment  is  desired  so  that  the  products  of  the  ITI-ALC 
Phase  2  effort  could  be  applicable  to  a  production  environment. 


1.2  Products 

The  methodology  used  started  by  considering  the  trade  study  space  of  available  wrapper 
technologies  available.  Candidate  technologies  were  gathered,  and  the  leading  contenders 
were  selected.  These  were  then  further  analyzed  to  determine  which  the  most  attractive. 

Initial  candidates  considered  included  Common  Object  Request  Broker  Architecture  (CORBA), 
Java  Remote  Method  Invocation  (RMI),  the  Open  Software  Foundation’s  Distributed 
Computing  Environment  (DCE),  and  proprietary  middleware  such  as  Prism  Openbase  and 
Microsoft’s  Distributed  Component  Object  Model  (DCOM).  The  decision  was  made  to 
concentrate  on  emerging  open  technologies  that  will  provide  a  wide  range  of  capabilities.  An 
emphasis  was  placed  on  standards-based  technology  with  wide  support.  Based  on  these 
criteria,  technologies  were  picked  for  further  analysis.  Commercially  available  packages  within 
those  technologies  were  selected  for  more  extensive  evaluation. 

The  initial  technology  downselect  was  made  on  the  basis  of  the  maturity  of  a  technology,  its 
prospects  for  the  future,  its  capabilities,  and  whether  it  was  based  on  an  open  architecture. 
Closed  architectures  like  Openbase  and  DCOM  were  rejected.  CORBA  was  considered  to 
supersede  DCE  since  CORBA  is  focused  Object  Oriented  technology.  CORBA  and  Java  RMI 
were  thus  considered  to  be  the  leading  candidates  on  the  basis  of  these  criteria.  The  CORBA 
specification  is  the  product  of  the  Object  Management  Group  (OMG)  consortium,  which 
includes  over  700  companies.  It  specifies  a  structure  through  which  objects  may  be  found  on 
a  wide  variety  of  platforms  and  transparently  executed.  CORBA-compliant  ORBs  on  the 
market  include  Digital’s  Object-Broker,  Visigenic’s  VisiBroker,  IBM’s  SOM,  Sun’s  NEO,  HP’s 
Orb  Plus,  Iona  Technologies’  Orbix,  Expersoft’s  Power-Broker,  and  ICL’s  DAIS.  Java  is  a 
language  and  execution  environment  developed  by  JavaSoft,  a  subsidiary  of  Sun.  Its 
compiled  code  is  executed  by  a  virtual  machine,  so  that  the  same  object  code  may  be 
executed  on  a  wide  variety  of  platforms.  RMI  is  a  mechanism  that  allows  invocation  of  Java 
objects  on  remote  platforms. 
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Both  CORBA  and  Java  RMI  strongly  support  modern  Object  Oriented  design  and  coding.  The 
CORBA  packages  picked  for  further  consideration  were  Iona  Technologies’  Orbix,  Visigenics’ 
VisiBroker,  and  Expersoft’s  CORBAplus.  These  were  picked  on  the  basis  of  market  share, 
maturity,  and  technical  quality. 


1.3  Environment 

The  platforms  on  which  these  packages  will  run  are  PC  clients,  UNIX  servers,  or  Windows  NT 
servers.  One  interfacing  legacy  system  runs  on  a  UNIX  platform  with  Oracle,  others  run  on  an 
IBM.  The  demonstration  environment  assumes  one  or  two  servers  and  at  most  fifteen  users 
using  at  most  five  clients.  The  full-up  environment  assumes  an  ALC  with  perhaps  1500  users, 
three  servers,  and  500  client  devices. 


1.4  Summary  of  Best  Candidate 

Of  the  two  technologies  analyzed,  CORBA  was  deemed  preferable:  it  has  a  broad  vendor 
base,  a  standard  that  is  managed  by  a  broad-based  consortium  by  hundreds  of  vendors,  and 
several  years  of  development.  Of  the  CORBA  packages,  Visigenics’  VisiBroker  appears  to  be 
the  best  candidate.  It  has  a  large  installed  base — it  is  embedded  into  Netscape’s 
Communicator  and  Enterprise  server.  Our  experience  is  that  it  was  the  easiest  to  use  and  a 
considerably  more  reliable  package  than  Orbix.  Visigenics  is  a  stable  company. 

The  packages  not  selected  provide  adequate  tools  for  straightforward  implementations  for  an 
ITI-ALC  demonstration.  As  reflected  in  the  trade  study  assessment  table,  Table  1,  however, 
they  have  deficiencies  that  make  them  less  desirable  than  VisiBroker. 

Our  experience  with  Orbix  was  that,  while  it  is  an  adequate  tool,  the  number  of  anomalies 
experienced  during  its  use  makes  it  an  undesirable  package,  particularly  since  Iona’s  technical 
support  organization  has  been  unresponsive. 

Expersoft’s  CORBAplus  is  a  technically  competent  package.  It  does  not,  however,  have  the 
market  share  enjoyed  by  Orbix  or  VisiBroker.  Expersoft  is  privately  held,  unlike  Visigenics, 
which  is  listed  on  NASDAQ.  This,  coupled  with  the  difficulty  encountered  in  contacting  their 
sales  organization,  suggests  that  Expersoft  is  less  likely  to  have  strong  long-term  vendor 
stability  than  Iona  or  Visigenics. 

RMI  is  appealing  if  the  sole  criterion  is  price.  It  lacks,  however,  the  breadth  of  industry 
participation  of  the  OMG,  the  body  that  establishes  the  CORBA  standard.  The  CORBA 
standard  has  been  in  development  since  1991,  Java  since  1994,  and  Java  RMI  has  been 
available  only  about  a  year.  It  is  possible  that  Java  RMI  will  become  a  major  industry 
standard,  but  while  Java  packages  are  beginning  to  appear  in  the  market  place,  they  are  not 
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yet  as  prevalent  as  packages  in  more  traditional  languages  such  as  C  or  C++.  RMI  also  lacks 
the  standardized  spectrum  of  services  that  are  rapidly  being  specified  by  OMG  and 
implemented  by  CORBA  vendors. 


1.5  Future  Considerations 

The  CORBA  and  Java  market  places  are  very  dynamic.  It  is  worthwhile  to  keep  track  of  the 
state  of  the  products  available,  since  the  current  leading  products  and  the  condition  and 
responsiveness  of  their  producing  companies  have  the  potential  for  swift  change. 


2.0  Trade  Study  Information 

Three  CORBA  ORBs — Orbix,  VisiBroker,  CORBAplus — have  been  installed  and  exercised  at 
the  Advanced  Technology  Laboratories.  They  were  generally  easy  to  install.  Of  the  three 
packages,  VisiBroker  was  the  easiest  to  use.  Our  experience  with  Orbix  has  been  that  there 
are  many  bugs  in  the  implementation,  and  that  the  quality  of  technical  support  was  not  good. 
For  example,  the  initial  Orbix  implementation  of  the  Internet  InterORB  Protocol  (MOP)  would 
not  run  at  all,  its  successor  ran  but  not  properly,  and  while  the  current  implementation  is 
supposed  to  run  correctly,  we  have  not  been  able  to  verify  this  since  another  anomaly  in  Orbix 
prevents  our  test  program  from  running  properly. 

All  three  CORBA  implementation  have  industrial  scale  implementations — they  should  provide 
adequate  throughput  for  an  ITI-ALC  demonstration  or  an  implementation  of  the  ITI-ALC 
capabilities  in  a  production  environment.  Java  RMI  is  new,  so  there  is  no  installed  base  of 
large-scale  production  systems.  It  should  be  adequate,  however,  for  an  ITI-ALC 
demonstration. 


3.0  Trade  Study  Assessment  Table 


3.1  Evaluation  Criteria 

For  each  of  the  downselected  candidate  packages,  a  Kempner-Tregoe  trade  study  was 
executed.  This  entails  developing  a  set  of  evaluation  criteria.  The  ITI-ALC  team  reviewed  the 
criteria,  and  their  comments  were  addressed  by  modifications  to  the  criteria.  The  criteria  were 
grouped,  and  each  group  of  criteria  was  assigned  a  weight.  Each  criterion  in  each  group  was 
assigned  an  intra-group  weight.  From  these,  an  overall  weight  was  developed  for  each 
criterion,  using  the  algorithm  described  in  Supplement  A-6-2.  This  method  of  developing 
criterion  weights  was  used  so  that  the  criteria  groups  would  be  weighted  as  desired.  It  avoids 


189 


inadvertently  giving  a  criterion  group  too  much  weight  because  many  there  are  many  criteria  in 
it,  or  giving  a  criterion  group  too  little  weight  because  there  are  few  criteria  in  it.  The  resulting 
weights  are  those  that  appear  in  the  criterion  weights  in  Supplement  A-6-1  and,  with  a 
modification  discussed  in  Section  3.2,  in  Table  A-6-1. 

Evaluation  criteria  and  their  weights  were  created  and  evaluated  from  two  perspectives.  One 
was  the  applicability  of  the  candidate  technology  or  product  to  fulfill  the  demonstration  needs  of 
the  ITI-ALC  program  over  its  three-year  life.  The  other  was  the  applicability  to  a  production 
version  of  the  capabilities  demonstrated  during  the  ITI-ALC  program.  The  latter  view  was 
driven  by  the  desire  to  make  whatever  is  developed  in  the  ITI-ALC  program  as  reusable  as 
possible.  This  has  implications  in  evaluating  the  corporate  strength  of  the  vendor,  the 
scalability  of  the  products,  and  its  cost  structure. 

The  evaluation  criteria  are  described  in  Supplement  A-6-1,  with  the  weights  assigned  to  each 
criterion.  Producibility  was  not  included  as  a  criterion,  since  it  is  not  applicable  to  middleware. 

3.2  Trade  Study  Assessment  Table 

Table  A-6-1  is  the  Trade  Study  Assessment  Table  for  the  products.  This  table  presents  the 
evaluation  criteria,  which  are  described  in  more  detail  in  Supplement  A-6-1 .  It  gives  the  weight 
for  each  criterion,  and  a  raw  score  for  each  product  and  criterion.  From  these,  a  weighted 
score  for  each  product  and  criterion  is  derived,  the  product  of  the  criterion  weight  and  the  raw 
product  score.  The  weighted  scores  for  each  product  are  summed  to  provide  the  overall 
measure  for  the  product.  The  rationale  for  the  assignment  of  raw  scores  for  each  criterion  is 
given  below. 

Capabilities 

Language  compatibility 

The  principal  languages  to  be  used  in  the  development  of  the  ITI-ALC  prototypes  are  taken  to 
be  C++  and  Java.  Databases  may  also  be  accessed  through  SQL.  All  three  CORBA 
packages  offer  compatibility  with  these  languages  as  well  as  others.  Java  RMI  provides  cross¬ 
platform  only  in  a  Java  environment,  although  SunSoft  will  support  the  Internet  InterORB 
Protocol  (MOP)  to  allow  invoking  objects  in  other  languages.  The  three  CORBA  products 
accordingly  received  a  higher  raw  score. 

Platform  compatibility 

The  platforms  to  be  used  for  development  of  the  ITI-ALC  prototypes  are  taken  to  be  a  subset 
of  UNIX,  Windows  NT,  Windows  95,  and  possibly  Windows  CE  for  clients.  All  of  the  four 
packages  evaluated  are  compatible  with  these  platforms,  with  the  possible  exception  of 
Windows  CE. 
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Supporting  product  availability 

All  capabilities  and  functionality  provided  by  Java  are  available  in  the  CORBA  environment. 
The  three  CORBA  packages  in  addition  provide  a  suite  of  standardized  services  whose 
specification  is  managed  by  the  OMG.  The  three  CORBA  products  accordingly  receive  a 
higher  raw  score. 
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Table  A-6-1  Middleware  Trade  Study  Assessment 


Capabilities 

Language  compatibility 

Platform  compatibility 

Supporting  product  availability 

Standards  based 

Flexibility 

Distribution 

Scalability 

Ease  of  Use 

Leaming.curve 
implementation  ease 

Performance 

Throughput  rate 
Time  to  make  connection 
Maximum  connections 
Time  to  bring  up  applications 
Memory  needed 

Reliability 

Product  maturity 
Reliability 
Installed  base 
Vendor  stabilitiy 

Cost 

Acquisition  cost 
Maintenance  cost 

Administration 

Ease  of  installation 
Ease  of  management 


|  Raw  Product  Scores 

Weighted  Product  Scores 

Visi 

CORBA 

Java 

Visi 

CORBA 

Java 

wt 

Orbix 

Broker 

plus 

RMI 

Orbix 

Broker 

plus 

RMI 

5 

10 

10 

10 

5 

50 

50 

50 

25 

5 

10 

10 

10 

10 

50 

50 

50 

50 

4 

8 

7 

8 

6 

32 

28 

32 

24 

5 

10 

10 

10 

7 

50 

50 

50 

35 

5 

10 

8 

10 

7 

50 

40 

50 

35 

4 

8 

8 

8 

10 

32 

32 

32 

40 

3 

9 

9 

9 

5 

27 

27 

27 

15 

7 

6 

8 

7 

9 

42 

56 

49 

63 

7 

7 

8 

7 

8 

49 

56 

49 

56 

25 

6 

8 

8 

8 

150 

200 

200 

200 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

n/a 

8 

8 

9 

9 

7 

64 

72 

72 

56 

6 

8 

9 

9 

9 

48 

54 

54 

54 

6 

10 

10 

8 

5 

60 

60 

48 

30 

8 

10 

9 

7 

10 

80 

72 

56 

80 

5 

8 

8 

8 

10 

40 

40 

40 

50 

8 

8 

8 

8 

10 

64 

64 

64 

80 

10 

8 

8 

8 

8 

80 

80 

80 

80 

|10 

7 

8 

8 

8 

70 

80 

80 

80 

TOTAL  WEIGHTED  SCORES  1 

1038 

1111 

1083 

1053 

Orbix 

Visi 

CORBA 

Java 

Broker 

plus 

RMI 
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Standards  based 

The  three  CORBA  packages  are  based  on  a  standard  supported  by  a  broad  consortium  of 
commercial  software  houses.  Java  RMI  is  a  product  of  a  single  vendor,  the  SunSoft  subsidiary 
of  Sun  Microsystems.  The  three  CORBA  products  accordingly  receive  a  higher  raw  score. 

Flexibility 

The  three  CORBA  packages  implement  the  extremely  flexible  environment  provided  by  the 
CORBA  specification.  They  provide  dynamic  discovery  of  objects  and  subsequent  invocation, 
wire-level  transaction,  persistent  object  references,  and  multi-language  object  invocation.  The 
three  CORBA  products  accordingly  receive  a  higher  raw  score. 

Distribution 

The  CORBA  environment  is  more  complex  for  implementation  and  operation.  Java  RMI 
accordingly  receives  a  higher  raw  score. 

Scalability 

An  implementation  of  the  capabilities  of  the  ITI-ALC  program  at  an  Air  Logistics  Center  would 
involve  1000  to  2000  mechanics  and  support  personnel.  The  CORBA  packages  under 
consideration  have  successful  deployments  this  size  and  larger.  The  three  CORBA  products 
accordingly  receive  a  higher  raw  score. 

Ease  of  Use 

Learning  curve 

Of  the  three  CORBA  implementations,  the  easiest  to  learn  was  VisiBroker.  The  constructs  it 
provides  fit  well  with  C++  constructs.  Orbix  requires  a  more  complex  runtime  environment, 
and  CORBAplus  makes  use  of  macros  that  obscure  what  is  happening.  Java  RMI  is  easier  to 
learn. 

Implementation  ease 

Implementation  within  the  CORBA  packages  is  easiest  for  VisiBroker.  Orbix  is  the  least 
convenient  for  setting  up  an  application  due  to  the  complexity  of  its  run  time  environment. 
CORBAplus  provides  powerful  but  somewhat  arcane  command  line  constructs.  Java  RMI  is 
easy  to  use  because  of  its  simpler  environment. 

Performance 

Of  the  detailed  criteria  in  this  category,  measurements  were  available  only  for  throughput.  The 
weights  of  the  criteria  in  this  criterion  group  were  correspondingly  adjusted  by  putting  all  of  the 
weight  on  the  throughput  criterion.  In  throughput,  Orbix  was  noticeably  slower  than  the  other 
two  CORBA  packages. 
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Reliability 

Product  Maturity 

Of  the  CORBA  packages  Orbix  is  the  oldest.  Since  our  experience  at  the  Advanced 
Technology  Laboratories  with  Orbix  is  that  its  releases  have  an  untoward  number  of 
implementation  anomalies,  Orbix  must  be  considered  a  mixture  of  mature  and  immature 
implementation,  and  is  correspondingly  rated  lower  than  the  other  CORBA  packages.  Java 
RMI  is  a  new  product;  it  is  in  production,  but  it  does  not  have  the  time  in  the  market  of  the 
CORBA  packages. 

Product  Reliability 

Our  experience  with  Orbix  has  been  that  it  is  unreliable.  In  addition  to  poor  reliability,  the  Iona 
technical  support  has  been  unresponsive  and  poor,  so  problems  with  Orbix  have  been  slow  to 
be  resolved.  We  have  had  less  experience  with  VisiBroker,  CORBAplus,  and  RMI,  but  have 
not  experienced  the  frequent  problems  with  them  that  we  encountered  with  Orbix. 

Installed  Base 

Orbix  has  the  largest  major  installed  base.  VisiBroker  also  has  a  substantial  base,  and 
VisiBroker  2.5  is  embedded  in  the  Netscape  Communicator  and  Enterprise  server. 
CORBAplus  has  a  smaller  installed  base,  and  Java  RMI  is  a  new  product. 

Vendor  Stability 

Iona  Technologies  is  the  largest  CORBA  package  vendor  and  may  be  considered  to  be  stable. 
Visigenic  is  a  public  company  listed  on  the  NASDAQ  with  a  stable  price  and  may  be 
considered  to  be  stable.  Expersoft  is  privately  held.  Our  experience  in  contacting  the 
Expersoft  marketing  organization  suggests  that  the  company  is  undergoing  some  degree  of 
reorganization  and  at  the  same  time  it  is  seeking  additional  capital.  Its  CORBAplus  product  is 
accordingly  down  rated.  JavaSoft  is  a  Sun  Microsystems  subsidiary,  and  may  be  considered 
very  stable. 

Cost 

Acquisition  Cost 

Orbix  structures  its  development  licensing  cost  on  the  basis  of  development  platforms  and 
developers.  The  larger  of  the  number  of  processors  and  the  number  of  developers  is  the 
number  of  licenses  required.  Each  development  license  on  a  UNIX  platform  costs  $5000;  on  a 
Windows  NT,  Windows  95,  Macintosh,  or  OS/2  it  costs  $2500.  Run  time  licenses  for  all 
platforms  are  $100.  Technical  support  for  UNIX  is  $750,  for  Windows  $400.  Major  upgrades 
are  available  at  20%  of  list  price. 

CORBAplus  development  licenses  are  priced  separately  for  Java  and  C++.  They  cost  $2995 
per  C++  UNIX  seat  and  $1995  per  C++  seat.  This  includes  their  Naming  Service  and 
Relationship  Service,  Life  Cycle  Service,  and  a  collection  of  Rogue  Wave  tools.  The  current 
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Java  development  seats  are  free;  however  their  OMG  compliant  IDL  compiler  supporting  JDK 
1.1.2  that  will  be  available  soon  will  be  $799  per  seat,  for  either  platform.  Technical  support  for 
the  C++  licenses  is  $450  per  annum  for  UNIX,  for  either  platform.  Technical  support  for  the 
C++  licenses  are  $450  per  annum  for  UNIX,  $300  for  NT,  and  for  Java  $150  for  either  platform. 
Run  time  licenses  are  $3595  per  UNIX  processor  and  $2595  per  NT  processor;  support  is 
respectively  $525  and  $325  per  annum.  Other  services:  Events  is  $895  on  UNIX  and  $595  on 
NT;  a  partial  implementation  of  Persistence  is  $1295  on  UNIX  and  $995  on  NT. 

VisiBroker  development  seat  licenses  are  $2995  for  UNIX,  $1995  for  Windows  NT  or  95  for 
either  C++  or  Java.  Seats  for  both  are  $3995  for  UNIX,  $2995  for  Windows  NT  or  95. 
Runtime  license  pricing  is  based  on  the  number  of  processors,  it  is  $2400  per  processor  for 
UNIX  and  $1665  for  NT. 

All  the  CORBA  vendors  offer  modest  volume  discounts. 

Cost  does  not  seem  to  be  a  discriminator  among  the  three  CORBA  packages,  especially  on 
the  scale  of  the  ITI-ALC  Phase  2  contract  of  an  acquisition  contract.  Consider  the  following 
development  and  deployment  models  for  deriving  comparative  cost.  (Note  that  these  are  not 
intended  to  reflect  the  actual  configuration  acquired.  They  are  for  comparative  purposes  only.) 

Using  a  model  for  the  Phase  2  ITI-ALC  development  of  3  C++  seats  on  NT,  3  servers  for  3 
years  and  5  clients,  all  three  packages  show  a  cost  of  1 1  to  12  thousand  dollars. 

Using  a  model  for  a  deployment  development  of  2  C++  seats  and  2  Java  seats,  one  UNIX  and 
one  NT  for  each,  2  UNIX  servers  and  2  NT  servers  and  20  clients,  all  three  packages  show  a 
cost  of  15  to  25  thousand  dollars. 

Using  a  model  of  5  UNIX  servers  and  10  NT  servers,  all  three  packages  show  a  deployment 
cost  of  about  60  thousand  dollars. 

Java  RMI  is  available  from  JavaSoft  as  part  of  its  JDK  1.1  product,  which  may  be  downloaded 
for  free.  Source  code  for  JDK  1.1  requires  a  licensing  fee,  but  should  not  be  needed  for 
development  work.  Java  RMI  clearly  has  the  lowest  cost,  but  it  may  not  remain  free 
indefinitely,  since  it  is  reasonable  to  expect  that  JavaSoft  will  in  the  future  try  to  profit  from  its 
Java  development,  either  by  charging  for  Java  or  by  making  it  necessary  to  buy  ancillary 
software  products. 

Administration 

Ease  of  installation 

Ease  of  installation  is  roughly  equivalent  for  all  four  packages. 
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Ease  of  Management 

Among  the  CORBA  packages,  VisiBroker  is  the  easiest  package  to  use  when  setting  up  and 
maintaining  a  runtime  environment.  It  needs  only  one  daemon  per  network — any  client  will  find 
it.  Orbix,  in  comparison,  requires  a  daemon  on  each  node  on  which  there  exists  a  server 
object,  as  well  as  several  configuration  files  and  environment  variables.  CORBAplus  and  Java 
RMI  ease  of  use  are  comparable  to  that  of  VisiBroker,  albeit  slightly  less. 


4.0  Product  Experience 

The  CORBA  packages  were  installed  and  exercised  at  ATL.  They  were  all  adequate  for 
straightforward  applications.  The  most  striking  result  was  the  difficulty  that  was  experienced  in 
resolving  anomalies  in  Orbix  with  Iona  Technologies’  technical  support  organization. 


5.0  Source  Summary 

The  information  on  which  the  evaluation  was  based  was  drawn  from  a  variety  of  sources. 
Material  from  the  Web  sites  of  all  four  vendors  and  other  vendors  such  as  Netscape  provided 
much  technical  information  about  the  packages  under  evaluation.  The  vendors’  sales 
organizations  and  technical  personnel  were  also  contacted.  Other  information  was  drawn  from 
texts:  Instant  CORBA  by  Orfali,  Harkey,  and  Edwards,  and  CORBA  Fundamentals  and 
Programming,  by  Siegel.  The  Advanced  Technology  Laboratories  have  several  people 
conversant  with  CORBA  that  were  consulted  for  their  expertise  and  experience  in  implentation 
of  CORBA-based  systems  and  CORBA  packages,  notably  Dr.  Russel  Noseworthy,  Dr. 
Gautam  Thaker,  and  Dr.  Charles  Peck. 
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Supplement  A-6-1  Middleware  Trade  Study  Criteria 


CRITERIA 

| 

DESCRIPTION 

Language  compatibility 

5 

1  Compatible  with  a  single  language 

10  Compatible  with  all  the  languages 
expected  to  be  used  for  ITI-ALC:  C++, 

C,  Java,  Java  Script,  Perl,  SQL 

Platform  compatibility 

5 

1  Compatible  with  a  single  platform 

10  Compatible  with  the  platforms 

expected  to  be  used  for  ITI-ALC:  Sun, 
Windows  95,  Windows  NT,  Windows 

CE,  and  provides  complete  platform 
transparency 

Supporting  product  availability 

1 

1  No  third  party  vendors  write  for  it 

10  A  wide  variety  of  add-ons  are  available 
for  all  platforms  and  O/S’s  used  in  ITI- 
ALC 

Standards  based 

5 

1  Proprietary  environment 

10  Based  on  industry-wide  standards  and 
interoperable  with  comparable 
packages 

Flexibility 

5 

1  The  tool  provides  a  limited  set  of 
capabilities 

10  The  tool  provides  a  rich  environment 
with  a  broad  range  of  capabilities 

Distribution 

1  The  tool  requires  substantial  developed 
code  to  make  work  in  a  distributed 
environment 

10  The  tool  is  easy  to  install,  implement, 
and  operate  in  a  distributed 
environment.  Provisions  are  available 
for  dynamic  invocation  of  objects. 
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Scalability 

3 

Scalability  is  important  insofar  as  the  ITI- 
ALC  program  must  be  able  to  do  a 
reasonably  sized  demonstration,  and  to 
demonstrate  an  architecture  that  can  be 
scaled  to  an  operational  system.  The 
operational  system,  however,  may  of 
course  use  a  different  tool. 

1  The  tool  is  limited  to  an  implementation 
with  no  more  than  5  nodes  or  5  users. 

10  The  tool  is  capable  of  handling  200 
nodes  with  1 000  users. 

Learning  curve 

1  Interfaces  and  underlying  structures 
are  complex  and  require  at  least  a 
week’s  training  and  two  months 
familiarization 

10  Interfaces  are  intuitive  and  can  be 
mastered  in  a  day 

Implementation  ease 

7 

1  Implementation  using  the  tool  is 

tedious.  At  least  3  days  are  required  to 
implement  each  interface,  after  design 
is  complete 

10  Interfaces  are  easy  to  implement. 

Tools  provide  a  means  to  implement 
an  interface  in  less  than  an  hour. 

Throughput  rate 

5 

Varies  with  class  of  product 

Time  to  establish  connection 

5 

Varies  with  class  of  product 

Maximum  number  of  connections 

5 

Varies  with  class  of  product 

Time  to  bring  up  applications 

5 

Memory  needed 

5 

1  Too  large  to  fit  on  portable  devices:  >  5 
Mb 

10  Fits  on  all  devices -<  0.5  Mb 

Product  maturity 

8 

1  in  beta 

10  In  wide  use  5  years 

Reliability 

6 

1  Most  users  report  problems 

10  No  known  problems 

Installed  base 

6 

1  Single  user  base 

1 0  500  or  more  users 

Vendor  stability 

8 

1  Vendor  in  business  less  than  a  year 

10  Vendor  in  business  5  years  or  more 
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Acquisition  cost 

5 

This  includes  the  cost  of  the  development 
seats  as  well  as  of  the  operational  software 

1  Over  $200K 

10  Freeware 

Maintenance  cost 

8 

This  is  the  cost  of  annual  licenses  and 
upgrades 

1  Over  $50K 

10  Freeware 

Ease  of  installation 

10 

1  One  week  or  more  per  platform 

10  One  day  or  less  per  platform 

Ease  of  management 

10 

1  Requires  full  time  administrator 

10  Once  installed,  needs  no 
administration 
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Supplement  A-6-2  Criterion  Weight  Algorithm 


Consider  N  criterion  groups,  the  /th  with  weight  W,. 

Hi 

Let  the  /th  group  have  n-,  criteria,  the  yth  with  weight  wfl.  Let  ot  =  £w».  Let  M  be  the  highest 
possible  score. 

Let  a  product  score  s,y  on  the  yth  criterion  in  the  /th  group.  The  total  product  score  is  then 


,  Wi  Wij 

The  result  is  to  weight  each  criterion  by - 

Oi 


Finally,  find  the  biggest  weight  = 


max 


hj 


Wi  Wij 


v  J 


,  call  it  ju,  and  normalize  so  that  the 


corresponding  criterion  has  weight  10. 

Thus  weight  the  yth  criterion  in  the  /th  group  by 


10  Wi  wij 

/U  Oi 

and  round  to  an  integer  value. 
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This  spreadsheet  computes  overall  criterion  weights  from  criterion  group  weights 
and  criterion  weights  within  each  criterion  group  according  to  the  algorithm  above. 

It  is  the  source  of  evaluation  criterion  weights  from  Table  A-6-1 . 

max  weight  for  normalization::  1 0 


Criterion 

intra¬ 

group 

raw 

criterion 

normalized 

Groups  Criteria  Criterion  arouo  weiqhts 

weiqhts 

weiqhts 

weiqhts 

Capabilities  10 

Language  compatibility 

5 

1.613 

5 

Platform  compatibility 

5 

1.613 

5 

Supporting  product  availability 

4 

1.290 

4 

Standards  based 

5 

1.613 

5 

Flexibility 

5 

1.613 

5 

Distribution 

4 

1.290 

4 

Scalability 

3 

0.968 

3 

31 

Ease  of  Use  4 

Learning  curve 

5 

2.000 

7 

Implementation  ease 

5 

2.000 

7 

10 

Performance  8 

Throughput  rate 

4 

1.600 

5 

Time  to  establish  connection 

4 

1.600 

5 

Maximum  number  of  connections 

4 

1.600 

5 

Time  to  bring  up  applications 

4 

1.600 

5 

Memory  needed 

4 

1.600 

5 

20 

Reliability  8 

Product  maturity 

4 

2.286 

8 

Reliability 

3 

1.714 

6 

installed  base 

3 

1.714 

6 

Vendor  stability 

4 

2.286 

8 

14 

Cost  4 

Acquisition  cost 

4 

1.600 

5 

Maintenance  cost 

6 

2.400 

8 

10 

Administration  6 

Ease  of  installation 

3 

3.000 

10 

Ease  of  management 

3 

3.000 

10 

6 

max 

3.000 
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A-7.  Visualization  Trade  Study 
1.0  Summation 

1.1  Purpose 

The  purpose  of  this  trade  study  is  to  recommend  a  visualization  technology  for  use  in  developing  ITI- 
ALC  prototypes.  This  technology  will  provide  functionality  for  mechanics  to  display  and  annotate  a  real¬ 
time,  three-dimensional  model  of  an  airplane.  The  initial  intended  use  is  to  allow  inspectors  to  mark 
defects  directly  on  the  model  (i.e.  create  “hotspots”)  to  aid  in  communication  of  defect  location 
information  to  planners,  schedulers,  and  other  mechanics.  These  recorded  defects  could  also  provide 
the  data  to  construct  a  diagnostic  database. 

1.2  Products 

We  evaluated  the  following  products  in  this  trade  study: 


Company 

Product  Name 

Macromedia 

Director  6  Multimedia  Studio  (Table  A-7-1) 

Criterion  Software 

Renderware  2.1  (Table  A-7-3) 

Apple 

QuickTime  VR  2.0  and  Virtual  Tutor  for  QuickTime  VR  Bundle 
(QTVR)  (Table  A-7-2) 

Byte  by  Byte 

Soft  F/X  (Table  A-7-4) 

We  initially  considered  over  twenty  products.  Of  these,  we  selected  those  that  seemed  to  have  the  best 
ability  to  display  a  relatively  complex  model,  hilo.3ds,  on  a  mid-grade  Pentium  processor.  Hilo.3ds 
contains  116,841  polygons.  By  comparison,  the  legacy  F-15E  model  we  received  contained  over 
1,000,000  polygons.  We  also  looked  for  the  ability  to  include  hotspots. 

1.3  Environment 

Of  the  four  packages  in  the  downselect,  all  four  will  run  on  Windows  NT  and  Windows  95  platforms, 
although  QTVR  will  only  allow  development  on  a  Macintosh.  Only  Soft  F/X  will  not  run  on  a  Macintosh 
system.  Our  demonstration  environment  simulated  a  wearable  computer  attempting  to  manipulate  a 
detailed  three-dimensional  model  of  a  helicopter.  We  attempted  to  manipulate  the  test  model  in  various 
ways,  including  rotation  in  all  dimensions  and  the  addition  and  manipulation  of  hotspots. 

1.4  Summary  of  Best  Candidate 

We  cannot  recommend  any  of  the  four  products  reviewed.  See  section  1.5  below  for  more 
information. 

1.5  Evaluation  of  Other  Products/Tools 

None  of  the  four  technologies  are  adequate  for  our  needs.  All  of  them  are  too  slow  to  be 
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practical  for  use  by  mechanics,  and  most  are  also  unacceptable  for  other  reasons.  Director  6 
Multimedia  Studio  has  no  hotspot  support,  and  no  API  or  SDK.  Renderware  2.1  also  has  no  hotspot 
support,  although  hotspot  support  could  be  separately  coded  via  their  API.  QuickTime  VR  2.0  uses  an 
image-based  representation  of  the  airplane,  which  was  deemed  inferior  to  a  model-based 
representation.  It  is  also  considerably  slower  than  the  other  downselected  products.  Soft  F/X  also  has 
no  hotspot  support,  and  the  API  is  poor. 

1.6  Future  considerations 

While  performing  this  trade  study,  we  learned  that  VRML  (Virtual  Reality  Modeling  Language) 
appears  to  satisfy  all  functional  requirements.  We  suggest  that  further  research  be  done  to 
determine  whether  VRML  is  able  to  handle  the  requirements  put  forth  for  a  visualization 
package.  VRML  is  a  standard  accepted  by  many  large  organizations,  including  IBM,  Microsoft, 
Intel,  and  Netscape.  The  following  excerpt  is  taken  from  The  Annotated  VRML  2.0  Reference 
Manual  by  Rikk  Carey  and  Gavin  Bell: 

VRML,  sometimes  pronounced  'vermal’,  is  an  acronym  for  the  Virtual  Reality  Modeling 
Language.  Technically  speaking,  VRML  is  neither  virtual  reality  nor  a  modeling 
language.  Virtual  reality  typically  implies  an  immersive  3D  experience  (such  as  a  head- 
mounted  display)  and  3D  input  devices  (such  as  digital  gloves).  VRML  neither  requires 
nor  precludes  immersion.  Furthermore,  a  true  modeling  language  would  contain  much 
richer  geometric  modeling  primitives  and  mechanisms.  VRML  provides  a  bare  minimum 
of  geometric  modeling  features  and  contains  numerous  features  far  beyond  the  scope  of 
a  modeling  language. 

So  if  VRML  is  not  virtual  reality  or  a  modeling  language,  what  is  it?  There  are  several 
answers  to  this  question.  At  its  core,  VRML  is  simply  a  3D  interchange  format.  It  defines 
most  of  the  commonly  used  semantics  found  in  today's  3D  applications  such  as 
hierarchical  transformations,  light  sources,  viewpoints,  geometry,  animation,  fog, 
material  properties,  and  texture  mapping.  One  of  the  primary  goals  in  designing  VRML 
was  to  ensure  that  it  at  least  succeeded  as  an  effective  3D  file  interchange  format. 

A  second  answer  is  that  VRML  is  a  3D  analog  to  HTML.  This  means  that  VRML  serves 
as  a  simple,  multiplatform  language  for  publishing  3D  Web  pages.  This  is  motivated  by 
the  fact  that  some  information  is  best-experienced  three  dimensionally,  such  as  games, 
engineering  and  scientific  visualizations,  educational  experiences,  and  architecture. 
Typically,  these  types  of  projects  require  intensive  interaction,  animation,  and  user 
participation  and  exploration  beyond  what  is  capable  with  a  page-,  text-,  or  image- 
based  format  (i.e. ,  HTML). 

Many  of  the  same  issues  that  apply  to  authoring,  converting,  and  presenting  lETMs  via  SGML 
also  apply  to  3D  models  and  VRML.  Just  as  authoring  tools  exist  for  lETMs,  many  tools  are 
available  to  author  VRML  worlds.  Conversion  of  legacy  data  is  also  an  issue  in  the  VRML 
world.  For  example,  the  Air  Force  has  supplied  the  ITI-ALC  project  with  F-15E  data  in  the 
IGES  format  -  tools  must  be  found  to  convert  this  data  to  VRML.  Likewise,  many  tools  exist 
for  presentation  -  often  these  tools  are  an  enhancement  to  a  standard  web  browser.  And,  just 
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as  you  can  code  directly  in  SGML,  you  can  also  code  directly  in  VRML. 

Two  web  sites  serve  as  starting  points  for  further  VRML  exploration:  http://www.vrml.org/ is  the 
home  page  for  “The  VRML  Consortium”,  the  standards  body  dedicated  to  defining  and 
promoting  VRML;  and  http://www.sdsc.edu/vrml/ is  the  home  page  for  “The  VRML  Repository”, 
a  complete  listing  of  VRML  tools. 

2.0  Test  Data 

The  primary  test  data  used  was  a  three-dimensional  attack  helicopter  model,  hilo.3ds, 
containing  1 16,841  polygons.  This  is  an  order  of  magnitude  less  than  legacy  data  given  to  us 
by  the  Air  Force  for  their  F-15E  fighter  jet. 
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3.0  Trade  Study  Assessment  Tables 


Table  A-7-1  Director  6  Multimedia  Studio 

Trade  Study  Criteria 

To  help  in  understanding  these  criteria,  consider  the  three  hardware  platforms  we  are 
contemplating 

using  the  visualization  software  on: 

•  A  wearable  computer  displaying  simplified  wire-frame  models. 

•  A  laptop  computer  on  the  “high-boy”  displaying  shaded  models. 

•  A  desktop  workstation  displaying  full  three-dimensional  models. 

7/8/97 

Criteria 

|Rqmts 

Weight 

Score 

Total 

Comments 

1.1  Curved  Surfaces 

The  visualization  tool  must  be 
able  to  draw  smooth,  curved 
surfaces  to  correctly  model  an 
aircraft. 

8 

1.2  Hotspots 

The  tool  must  be  able  to  associate 
data  with  locations  on  the  model. 
This  includes  the  ability  to  add, 
move,  and  remove  links 

10 

0 

0 

Press  releases 
suggested  hotspot 
capabilities,  which  did 
not  exist  in  the  product. 

1.3  3-D  Navigation 

The  tool  must  allow  navigation  of 
smooth  surfaces  of  the  model  in 
three  dimensions. 

10 

10 

100 

Since  an  airplane  exists 
in  three  dimensions, 
and  is  difficult  to 
represent  in  only  two, 
our  tool  must  be  able  to 
handle  three- 
dimensional  model 
representations. 
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Table  A-7-1  Director  6  Multimedia  Studio  (Continued) 


1.4  Manipulate  Multimedia  Data 

The  tool  must  be  able  to  associate 
data  such  as  sound,  video,  digital 
images,  and  text  added  by  the 
inspector  with  the  model.  The  data 
should  be  stored  in  a  format 
amenable  to  searching, 
presentation,  and  indexing. _ 


2.0  Non-functional  Requirements 
ALC 


2.1  Software  Architecture 


2.1.1  Client/Server  Architecture 

The  visualization  should  run  in  a 
client/server  environment  (using 
CGI  scripts,  etc.),  preferably  in  a 
distributed  object  environment 
(using  an  API). 


2.1.2  Web  Interface 

The  tool  should  be  easily 
intergratable  with  a  web  browser 
capable  of  navigating  3D  models 
(both  image-based  like  QuickTime 
VR  and  model-based,  such  as 
VRML).  By  using  a  browser- 
based  user  interface  the 
development  time  of  the  user 
interface  can  be  significantly 
reduced. 


s  Note:  all  requirements  come  from  the  Scenario:  ITI- 

RISS  section.  Non-functional  requirements  are  those 
criteria  that  are  needed  to  support  the  functional 
requirements _ 


In  the  case  of  the 
viewing  models  on  a 
wearable  computer,  the 
rendering  and  data 
manipulation  needed  to 
display  the  model 
could  be  performed  at 
the  server. 
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Table  A-7-1  Director  6  Multimedia  Studio  (Continued) 

2.2  Development  Environment  _ _ _ _ ^ _ 

2.2.1  Development  Environment  9  0  0 

Platform  Independence 

Any  visualization  code  to  be 
written  should  be  in  Java  because 
of  its  high  platform  independence. 

Or,  the  visualization  code  should 
be  accessible  from  Java  via  an 

API. _ ^ _ I _  1 _ 

2.3  Target  Environment 


2.3.1  Target  Platform  Browsing 

The  target  platform  must  include 
software  components  that  allow 
the  browsing  of  image-based 
(QuickTime  VR)  or  model-based 
(VRML)  representations. 

10 

8 

80 

Model-based 

representations 

2.3.2  Target  Platform  Authoring 

The  target  platform  must  include 
software  components  that  allow 
the  creation  and  editing  of 
visualization  models. 

1 

10 

10 

2.3.3  Target  Platform 
Independence.  It  is  desirable  that 
the  tool  is  independent  of  a 
specific  target  platform 

5 

7 

35 

Windows  and 
Macintosh 

|2.4  Performance 

2.4.1  Response  Time 

The  tool  must  be  able  to  handle 
representing  complex  models 
quickly.  Any  tool  with  very 
noticeable  lag  (greater  than  1  sec) 
when  navigating  the  model  must 
be  rejected. 

7 

0 

0 

As  a  test  model,  we 
suggest  using  hilo.3ds, 
a  2.28  MB  helicopter 
model,  and  run  a 
visualization  of  it  on  a 
486  PC  with  66  MHz 
16  MB  of  RAM.  This 
is  similar  to  a  low-end 
wearable  computer. 
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Table  A-7-1  Director  6  Multimedia  Studio  (Continued) 


2.5  Data  format 


2.5.1  Model  Representation 

The  visualization  tool  must  use  a 
model  representation.  This  could 
be  either  image-based  (such  as 
QuickTime  VR)  or  CAD-based 
(such  as  VRML). 


3.0  Reliabili 


Not  applicable 


4.0  Maintainabili 


Not  Applicable 


5.0  Producibility  _ 


5.1  Low  Cost  -  The  cost  of  the 
tool  should  be  less  than  $1500 


6.0  Dependencies _ 

6.1  API  Support  by  HTTP 
Server  The  HTTP  server  should 
support  remote  method  invocation 
by  the  visualization  client  running 
on  the  target  platform. _ _ 


6.2  Bandwidth  The  visualization 
server  part  of  the  tool  should  run 
on  a  very  powerful  workstation  to 
allow  for  continuous  smooth 
animations  while  navigating 
through  the  model 


The  preference  for 
either  model 
representation  should 
be  determined  during 
the  trade  study  phase. 
CAD-based 


6 

6 

36 

$998 

_ 

10 

0 

0 

7 

0 

0 

The  server  should  be  at 
least  a  250  MHz 
Processor  with  4  GB 
Disk  storage,  256  MB 
RAM. 
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Table  A-7-2  QuickTime  VR  2.0 


Trade  Study  Criteria 


To  help  in  understanding  these  criteria,  consider  the  three  hardware  platforms  we  are 
contemplating 

using  the  visualization  software  on: 

•  A  wearable  computer  displaying  simplified  wire-frame  models. 

•  A  laptop  computer  on  the  “high-boy”  displaying  shaded  models. 

•  A  desktop  workstation  displaying  full  three-dimensional  models. _ 


7/8/97 


Criteria _ Rqmts  |Weight  | Score 

1.0  Functional  Requirements  N 

A 


1.1  Curved  Surfaces 

The  visualization  tool  must  be 
able  to  draw  smooth,  curved 
surfaces  to  correctly  model  an 
aircraft. 


1.2  Hotspots 

The  tool  must  be  able  to  associate 
data  with  locations  on  the  model. 
This  includes  the  ability  to  add, 
move,  and  remove  links 


1.3  3-D  Navigation 

The  tool  must  allow  navigation  of 
smooth  surfaces  of  the  model  in 
three  dimensions. 


1.4  Manipulate  Multimedia  Data 

The  tool  must  be  able  to  associate 
data  such  as  sound,  video,  digital 
images,  and  text  added  by  the 
inspector  with  the  model.  The  data 
should  be  stored  in  a  format 
amenable  to  searching, 
presentation,  and  indexing. _ 


100 

Many  programs  use  an 
insufficient  number  of 
polygons  to  represent  a 
curved  surface. 

100 

90 

Since  an  airplane  exists 
in  three  dimensions, 
and  is  difficult  to 
represent  in  only  two, 
our  tool  must  be  able  to 
handle  three- 
dimensional  model 
representations. 
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Table  A-7-2  QuickTime  VR  2.0  (Continued) 


2.0  Non-functional  Requirements  Note:  all  requirements  come  from  the  Scenario:  ITI- 
ALC 

RISS  section.  Non-functional  requirements  are  those 
criteria  that  are  needed  to  support  the  functional 
_ requirements _ _ 

2.1  Software  Architecture _ _ _ _ _ _ 

2.1.1  Client/Server  Architecture  I  10  0  0  In  the  case  of  the 

The  visualization  should  run  in  a  viewing  models  on  a 

client/server  environment  (using  wearable  computer,  the 

CGI  scripts,  etc.),  preferably  in  a  rendering  and  data 

distributed  object  environment  manipulation  needed  to 

(using  an  API).  display  the  model 

could  be  performed  at 

_ the  server. _ 

2.1.2  Web  Interface  5  10  50 

The  tool  should  be  easily 

intergratable  with  a  web  browser 
capable  of  navigating  3D  models 
(both  image-based  like  QuickTime 
VR  and  model-based,  such  as 
VRML).  By  using  a  browser- 
based  user  interface  the 
development  time  of  the  user 
interface  can  be  significantly 

reduced. _ _] _ | _ | _ | _ | _ 

2.2  Development  Environment 

9  [O  |0  I 


2.2.1  Development  Environment 
Platform  Independence 

Any  visualization  code  to  be 
written  should  be  in  Java  because 
of  its  high  platform  independence. 
Or,  the  visualization  code  should 
be  accessible  from  Java  via  an 
API. 
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Table  A-7 


2.3  Target  Environment 


2.3.1  Target  Platform  Browsing 

The  target  platform  must  include 
software  components  that  allow 
the  browsing  of  image-based 
(QuickTime  VR)  or  model-based 
(VRML)  representations. _ 

2.3.2  Target  Platform  Authoring 

The  target  platform  must  include 
software  components  that  allow 
the  creation  and  editing  of 
visualization  models. _ 

2.3.3  Target  Platform 
Independence.  It  is  desirable  that 
the  tool  is  independent  of  a 

specific  target  platform _ 

2.4  Performance 


2.4.1  Response  Time 

The  tool  must  be  able  to  handle 
representing  complex  models 
quickly.  Any  tool  with  very 
noticeable  lag  (greater  than  1  sec) 
when  navigating  the  model  must 
be  rejected. 


2.5  Data  format 


2.5.1  Model  Representation 

The  visualization  tool  must  use  a 
model  representation.  This  could 
be  either  image-based  (such  as 
QuickTime  VR)  or  CAD-based 
(such  as  VRML). 


3.0  Reliability 


Not  applicable 


-2  QuickTime  VR  2.0  (Continued) 


Image-based 

representation 


Macintosh  only 


Windows  and 
Macintosh 


As  a  test  model,  we 
suggest  using  hilo.3ds, 
a  2.28  MB  helicopter 
model,  and  run  a 
visualization  of  it  on  a 
486  PC  with  66  MHz 
16  MB  of  RAM.  This 
is  similar  to  a  low-end 
wearable  computer. 


The  preference  for 
either  model 
representation  should 
be  determined  during 
the  trade  study  phase. 
Image-based. 


4.0  Maintainability 


Not  Applicable 


Table  A-7-2  QuickTime  VR  2.0  (Continued) 

5.0  Producibility 

5.1  Low  Cost  -  The  cost  of  the 
tool  should  be  less  than  $1500 

6 

8 

48 

$400 

|6*0  Dependencies  I 

6.1  API  Support  by  HTTP 
Server  The  HTTP  server  should 
support  remote  method  invocation 
by  the  visualization  client  running 
on  the  target  platform. 

10 

0 

0 

6.2  Bandwidth  The  visualization 
server  part  of  the  tool  should  run 
on  a  very  powerful  workstation  to 
allow  for  continuous  smooth 
animations  while  navigating 
through  the  model 

7 

0 

0 

The  server  should  be  at 
least  a  250  MHz 
Processor  with  4  GB 
Disk  storage,  256  MB 
RAM.  " 

71 

482 
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Table  A-7-3  Renderware  2.1 


Trade  Study  Criteria 


To  help  in  understanding  these  criteria,  consider  the  three  hardware  platforms  we  are 
contemplating 

using  the  visualization  software  on: 

•  A  wearable  computer  displaying  simplified  wire-frame  models. 

•  A  laptop  computer  on  the  “high-boy”  displaying  shaded  models. 

•  A  desktop  workstation  displaying  full  three-dimensional  models. 


7/8/97 


Criteria 


1.0  Functional  Requirements 


1.1  Curved  Surfaces 

The  visualization  tool  must  be 
able  to  draw  smooth,  curved 
surfaces  to  correctly  model  an 
aircraft. 


1.2  Hotspots 

The  tool  must  be  able  to  associate 
data  with  locations  on  the  model. 
This  includes  the  ability  to  add, 
move,  and  remove  links 


1.3  3-D  Navigation 

The  tool  must  allow  navigation  of 
smooth  surfaces  of  the  model  in 
three  dimensions. 


|Rqmts  |Weight  |Score  [T  otal 


Total  Comments 


Note:  all  requirements  come  from  the  Scenario:  ITI- 
ALC  RISS  section.  Functional  requirements  are  those 


1.4  Manipulate  Multimedia  Data 

The  tool  must  be  able  to  associate 
data  such  as  sound,  video,  digital 
images,  and  text  added  by  the 
inspector  with  the  model.  The  data 
should  be  stored  in  a  format 
amenable  to  searching, 
presentation,  and  indexing. _ 


10 

100 

Many  programs  use  an 
insufficient  number  of 
polygons  to  represent  a 
curved  surface. 

2 

20 

Limited  support  of 
some  data  linkage. 

Most  hotspot 
implementation  would 
have  to  be  written. 

9 

90 

Since  an  airplane  exists 
in  three  dimensions, 
and  is  difficult  to 
represent  in  only  two, 
our  tool  must  be  able  to 
handle  three- 
dimensional  model 
representations. 

1 

7 

Format  not  amenable 
to  searching, 
presentation,  or 
indexing. 
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Table  A-7-3  Renderware  2.1  (Continued) 


2.0  Non-functional  Requirements  Note:  all  requirements  come  from  the  Scenario:  ITI- 
ALC 

RISS  section.  Non-functional  requirements  are  those 
criteria  that  are  needed  to  support  the  functional 
_ requirements _ 

2.1  Software  Architecture _ _ _ _ _ _ 

2.1.1  Client/Server  Architecture  10  0  0  In  the  case  of  the 

The  visualization  should  run  in  a  viewing  models  on  a 

client/server  environment  (using  wearable  computer,  the 

CGI  scripts,  etc.),  preferably  in  a  rendering  and  data 

distributed  object  environment  manipulation  needed  to 

(using  an  API).  display  the  model 

could  be  performed  at 

_ the  server. _ 

2.1.2  Web  Interface  5  0  0 

The  tool  should  be  easily 

intergratable  with  a  web  browser 
capable  of  navigating  3D  models 
(both  image-based  like  QuickTime 
VR  and  model-based,  such  as 
VRML).  By  using  a  browser- 
based  user  interface  the 
development  time  of  the  user 
interface  can  be  significantly 

reduced. _ | _ | _ | _ | _ | _ 

2.2  Development  Environment  _ _ _ _ _ . 

2.2.1  Development  Environment  9  5  45  C++  API 

Platform  Independence 

Any  visualization  code  to  be 
written  should  be  in  Java  because 
of  its  high  platform  independence. 

Or,  the  visualization  code  should 
be  accessible  from  Java  via  an 
API. 


214 


Table  A-7-3  Renderware  2.1  (Continued) 


2.3  Target  Environment 


2.3.1  Target  Platform  Browsing 

The  target  platform  must  include 
software  components  that  allow 
the  browsing  of  image-based 
(QuickTime  VR)  or  model -based 
(VRML)  representations. _ 

2.3.2  Target  Platform  Authoring 

The  target  platform  must  include 
software  components  that  allow 
the  creation  and  editing  of 
visualization  models. _ 

2.3.3  Target  Platform 

Independence.  It  is  desirable  that 
the  tool  is  independent  of  a 
specific  target  platform _ 


2.5.1  Model  Representation 

The  visualization  tool  must  use  a 
model  representation.  This  could 
be  either  image-based  (such  as 
QuickTime  VR)  or  CAD-based 
(such  as  VRML). 


10  10  100  Model-based 

representation 


10  10 


Windows  and 
Macintosh 


2.4  Performance 

2.4.1  Response  Time 

The  tool  must  be  able  to  handle 
representing  complex  models 
quickly.  Any  tool  with  very 
noticeable  lag  (greater  than  1  sec) 
when  navigating  the  model  must 
be  rejected. 

7 

0 

0 

As  a  test  model,  we 
suggest  using  hilo.Sds, 
a  2.28  MB  helicopter 
model,  and  run  a 
visualization  of  it  on  a 
486  PC  with  66  MHz 

16  MB  of  RAM.  This 
is  similar  to  a  low-end 
wearable  computer. 

Not  applicable 

4.0  Maintainability 

Not  Applicable 

|5.0  Producibility 
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Table  A-7-3  Renderware  2.1  (Continued) 

6.0  Dependencies 

6.1  API  Support  by  HTTP 
Server  The  HTTP  server  should 
support  remote  method  invocation 
by  the  visualization  client  running 
on  the  target  platform. 

10 

0 

0 

6.2  Bandwidth  The  visualization 
server  part  of  the  tool  should  run 
on  a  very  powerful  workstation  to 
allow  for  continuous  smooth 
animations  while  navigating 
through  the  model 

7 

0 

0 

The  server  should  be  at 
least  a  250  MHz 
Processor  with  4  GB 
Disk  storage,  256  MB 
RAM. 

64 

497 

216 


Table  A-7-4  SoftF/X 

Trade  Study  Criteria 

To  help  in  understanding  these  criteria,  consider  the  three  hardware  platforms  we  are 
contemplating 

using  the  visualization  software  on: 

•  A  wearable  computer  displaying  simplified  wire-frame  models. 

•  A  laptop  computer  on  the  “high-boy”  displaying  shaded  models. 

•  A  desktop  workstation  displaying  full  three-dimensional  models. 

7/8/97 

Criteria 

Rqmts 

Weight 

Score 

Total 

Comments  | 

1.0  Functional  Requirements 

Note:  all  requirements  com 
ALC  RISS  section.  Functi< 
criteria  that  come  directly  i 

e  from  the  Scenario:  ITI- 
mal  requirements  are  those 
rom  use  cases  (scenarios). 

1.1  Curved  Surfaces 

The  visualization  tool  must  be 
able  to  draw  smooth,  curved 
surfaces  to  correctly  model  an 
aircraft. 

10 

10 

Many  programs  use  an 
insufficient  number  of 
polygons  to  represent  a 
curved  surface. 

1.2  Hotspots 

The  tool  must  be  able  to  associate 
data  with  locations  on  the  model. 
This  includes  the  ability  to  add, 
move,  and  remove  links 

10 

0 

0 

Hotspot  support  must 
be  fully  written  by  us 
through  the  API. 
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Table  A-7-4  Soft  FIX  (Continued) 


1.3  3-D  Navigation 

The  tool  must  allow  navigation  of 
smooth  surfaces  of  the  model  in 
three  dimensions. 


Since  an  airplane  exists 
in  three  dimensions, 
and  is  difficult  to 
represent  in  only  two, 
our  tool  must  be  able  to 
handle  three- 
dimensional  model 
representations. _ 


1.4  Manipulate  Multimedia  Data 

The  tool  must  be  able  to  associate 
data  such  as  sound,  video,  digital 
images,  and  text  added  by  the 
inspector  with  the  model.  The  data 
should  be  stored  in  a  format 
amenable  to  searching, 

presentation,  and  indexing. _ 

2.0  Non-functional  Requirements 
ALC 


2.1  Software  Architecture 


2.1.1  Client/Server  Architecture 

The  visualization  should  run  in  a 
client/server  environment  (using 
CGI  scripts,  etc.),  preferably  in  a 
distributed  object  environment 
(using  an  API). 


s  Note:  all  requirements  come  from  the  Scenario:  ITI- 

RISS  section.  Non-functional  requirements  are  those 
criteria  that  are  needed  to  support  the  functional 
requirements  _ 


In  the  case  of  the 
viewing  models  on  a 
wearable  computer,  the 
rendering  and  data 
manipulation  needed  to 
display  the  model 
could  be  performed  at 
the  server. 
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Table  A-7-4  Soft  FIX  (Continued) 

2.1.2  Web  Interface 

The  tool  should  be  easily 
intergratable  with  a  web  browser 
capable  of  navigating  3D  models 
(both  image-based  like  QuickTime 
VR  and  model-based,  such  as 
VRML).  By  using  a  browser- 
based  user  interface  the 
development  time  of  the  user 
interface  can  be  significantly 
reduced. _ 

2.2  Development  Environment 
2.2.1  Development  Environment 
Platform  Independence 
Any  visualization  code  to  be 
written  should  be  in  Java  because 
of  its  high  platform  independence. 

Or,  the  visualization  code  should 
be  accessible  from  Java  via  an 
API. _ 

2.3  Target  Environment 


2.3.1  Target  Platform  Browsing 

The  target  platform  must  include 
software  components  that  allow 
the  browsing  of  image-based 
(QuickTime  VR)  or  model-based 
(VRML)  representations. 

10 

10 

100 

Model-based 

representation 

2.3.2  Target  Platform  Authoring 

The  target  platform  must  include 
software  components  that  allow 
the  creation  and  editing  of 
visualization  models. 

1 

10 

10 

2.3.3  Target  Platform 
Independence.  It  is  desirable  that 
the  tool  is  independent  of  a 
specific  target  platform 

5 

5 

25 

Windows 
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Table  A-7-4  Soft  F/X  (Continued) 


2.4  Performance 


2.4.1  Response  Time 

The  tool  must  be  able  to  handle 
representing  complex  models 
quickly.  Any  tool  with  very 
noticeable  lag  (greater  than  1  sec) 
when  navigating  the  model  must 
be  rejected. 


2.5  Data  format 


2.5.1  Model  Representation 

The  visualization  tool  must  use  a 
model  representation.  This  could 
be  either  image-based  (such  as 
QuickTime  VR)  or  CAD-based 
(such  as  VRML). _ 


3.0  Reliabili 


Not  applicable 


4.0  Maintainabili 


Not  Applicable 


5.0  Producibili 


5.1  Low  Cost  -  The  cost  of  the 
tool  should  be  less  than  $1500 


6.0  Dependencies 


6.1  API  Support  by  HTTP 
Server  The  HTTP  server  should 
support  remote  method  invocation 
by  the  visualization  client  running 
on  the  target  platform. _ 


6.2  Bandwidth  The  visualization 
server  part  of  the  tool  should  run 
on  a  very  powerful  workstation  to 
allow  for  continuous  smooth 
animations  while  navigating 
through  the  model  _ 


As  a  test  model,  we 
suggest  using  hilo.3ds, 
a  2.28  MB  helicopter 
model,  and  run  a 
visualization  of  it  on  a 
486  PC  with  66  MHz 
16  MB  of  RAM.  This 
is  similar  to  a  low-end 
wearable  computer. 


9 

10 

90 

The  preference  for 
either  model 

representation  should 
be  determined  during 
the  trade  study  phase. 
Model-based. 

0 

0 

0 

0 

64 

494 

The  server  should  be  at 
least  a  250  MHz 
Processor  with  4  GB 
Disk  storage,  256  MB 
RAM. 
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4.0  Product  Experience 

The  modeling  programs  were  installed  and  exercised  rigorously.  For  each  program,  we  observed  the 
model  in  both  wireframe  and  rendered  modes.  We  observed  the  refresh  time  and  display  quality  (for 
rendered  mode).  The  programs  were  found  to  be  incapable  of  handling  models  at  the  required  detail 
level  in  reasonable  time.  We  next  attempted  to  implement  a  hotspot  for  the  model,  using  both  the 
standard  interface  and  the  provided  API.  This  testing  sufficed  to  show  that  none  of  the  technologies 
considered  would  be  adequate  for  our  needs.  We  also  looked  for  the  quality  of  the  web  interface  for 
each  of  these  products,  and  discovered  that  none  of  them  supported  any  web  interface. 

In  addition  to  the  formal  testing,  several  informal  tests  involving  smaller  models  were  employed.  These 
showed  that  all  of  the  programs  ran  acceptably  on  Pentium-class  machines.  However,  the  lack  of  an 
API  for  Director  6  Multimedia  Studio  made  it  unacceptable,  even  for  smaller  models.  Also,  the  Soft  F/X 
API  was  very  small,  requiring  large  amounts  of  code  to  be  generated  for  even  relatively  trivial  tasks. 

5.0  Source  Summary 

The  information  on  which  the  evaluation  was  based  was  drawn  from  a  variety  of  sources.  Material  from 
the  Web  sites  of  all  four  vendors  and  other  vendors  such  as  Netscape  provided  much  technical 
information  about  the  packages  under  evaluation.  The  vendors’  sales  organizations  and  technical 
personnel  were  also  contacted. 


Company 

Product  Name  /  Contact 

Macromedia 

Director  6  Multimedia  Studio 
http://www.macromedia.com/software/director/ 

Criterion  Software 

Renderware  2.1 

http://www.csl.com/RenderWare/ 

Apple 

QuickTime  VR  2.0  and  Virtual  Tutor  for  QuickTime  VR  Bundle 
http://quicktimevr.apple.com/ 

Byte  by  Byte 

Soft  F/X 

http://bvtebvbvte.com/sfxhome.htm 
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A -8  Miniature  Video  Camera  Trade  Study 
1.0  Summation 
1.1  Purpose 

The  purpose  of  this  trade  study  is  to  investigate  small  video  cameras  that  could  be  mounted  on 
a  head  worn  display  to  transmit  live  video  from  a  mechanic  working  on-  a  plane  to  a  remote 
expert.  The  original  intent  was  to  use  these  cameras  in  a  help  desk  scenario  or  engineering 
collaboration  scenario.  A  secondary  purpose  is  to  evaluate  the  capability  of  these  same 
cameras  to  capture  still  images.  In  constructing  the  evaluation  criteria,  we  assumed  the 
cameras  would  need  to  be  small  and  lightweight  for  use  with  a  head  worn  display.  We 
assumed  that  the  bandwidth  of  the  wireless  transmission  would  be  the  limiting  factor  for  video 
quality,  and  that  the  video  quality  of  the  cameras  would  need  to  be  at  least  as  good  as  allowed 
by  the  wireless  transmission  bottleneck. 


1.2  Products 

We  evaluated  the  following  products: 


Company 

Product  Name 

VLSI  Vision  Limited 

WL  5402  Lensed  Module  (Table  A-8-2) 

Marshall 

V-1247  Color  Camera  (Table  A-8-3) 

Marshall 

VI 207  B&W  camera  (Table  A-8-4 

Marshal 

V1207-PL  B&W  camera  with  pinhole  lens  (Table 
A-8-1) 

Marshall  decided  not  to  produce  the  V1210-SCT  miniature  camera  that  was  originally  to  be 
part  of  the  trade  if  it  became  available,  therefore  only  the  above  four  cameras  were  part  of  the 
trade.  They  include  one  color  camera,  one  CMOS  camera  -  this  is  the  first  camera  available  of 
this  new  breakthrough  technology  (all  the  others  are  charge  coupled  device  (CCD)  cameras),  a 
small  back  and  white  CCD  camera,  and  a  similar  camera  with  a  pinhole  lens.  Together  these 
cover  a  broad  spectrum  of  kinds  of  miniature  cameras  and  technologies.  While  three  of  the 
four  cameras  are  from  the  same  manufacturer,  we  considered  a  much  broader  range  of 
companies  and  cameras  before  downselecting  to  these.  At  the  time  of  the  downselect, 
Marshall  offered  the  smallest  black  and  white  and  color  cameras  available  for  CCD  cameras. 
An  unbiased  camera  expert  at  CMU  reviewed  our  choices  and  assured  us  they  represented  a 
good  evaluation  set.  Marshall  can  be  contacted  at  (310)  390-6608  (phone)  or  (310)  391-8926 
(fax).  Their  address  is  Marshall  Electronics  Inc.  P.O.  Box  2027,  Culver  City,  CA  90231.  VLSI 
Vision  LTD.  can  be  contacted  at  (408)  374-5323  (phone)  or  (408)  374-4722  (fax).  Their 
address  is  18805  Cox  Avenue,  Suite  260,  Saratoga,  CA  95070. 
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1.3  Environment 

To  test  the  cameras,  we  first  constructed  an  adapter  to  provide  power  and  output  suitable  for 
use  with  a  TV  monitor  or  computer.  A  schematic  in  included  below.  The  adapter  was 
constructed  from  the  following  materials: 

1-  Circuit  Board  and  Box 
1-  3  Pin  Connector 
1-  6  Pin  Connector 

1-  9v  300mA  AC  Wall  Cube 

2-  RCA  Jack  Cables 

The  power  source  (VCC,  Ground)  connects  to  central  pins.  Wires  run  from  those  pins  to 
camera  jumper  pins.  This  enables  a  compact  and  simple  design  for  adaptation  of  various 
miniature  camera  models.  Each  camera  is  wired  such  that  a  simple  "one  way  only"  jumper 
dictates  which  camera  plugs  into  a  jumper,  ensuring  that  polarity  and  audio\video  pins  connect 
correctly.  The  RCA  Jacks  are  wired  to  central  pins  and  connect  to  the  appropriate  camera 
jumper  pins  as  well.  These  cables  are  applicable  to  common  TVWCR  jacks  as  well  as  certain 
mobile  computing  devices.  The  power  source  connection  could  easily  be  modified  to 
accommodate  a  DC  power  source  (e.g.  a  9V  battery  for  mobile  use). 

We  connected  the  video  out  jack  from  the  cameras  up  to  a  MRT  micro's  VideoPort 
Professional  framegrabber  (www.mrtmicro.com),  which  digitizes  the  images  and  sends  them  to 
a  laptop  computer  to  display  in  640x480  resolution.  Note  that  the  VGA  resolution  is 
independent  of  the  resolution  of  the  video  cameras,  which  range  from  220  to  380  TV  lines 
(reported).  We  used  an  official  test  chart  to  compare  the  resolution  and  quality  of  the  images, 
and  evaluated  them  under  conditions  of  high  and  low  light. 


1.4  Summary  of  Best  Candidate 

Of  the  cameras  we  evaluated,  the  Marshall  V1207-PL  B&W  camera  with  pinhole  lens  is  clearly 
the  best  choice  for  use  with  a  head  worn  display.  It  has  the  highest  quality  image  of  the  four 
cameras  (see  section  II  for  a  detailed  analysis),  and  the  smallest  form  factor.  The  pinhole  lens 
gave  it  the  clearest  image.  Combining  these  features  with  its  moderate  power  consumption 
and  overall  robustness  to  different  environmental  conditions  gave  it  the  highest  score  of  921 .  It 
does  have  several  shortcomings  however.  The  pinhole  mount  makes  it  harder  to  attach  other 
lenses  if  desired  (unlike  the  Marshall  V-1207  and  the  other  cameras).  In  addition,  it  is  a  black 
and  white  camera  -  if  color  is  required  than  a  different  camera  should  be  used.  Color  CCD 
cameras  are  much  heavier  and  bulkier  in  general  and  as  a  result  are  marginal  for  head  worn 
use.  Light,  high  resolution,  color,  CMOS  cameras  should  soon  be  available  from  WL  and 
other  companies.  These  represent  the  greatest  hope  for  lightweight,  color,  and  head  worn 
cameras.  In  the  meantime,  of  the  reviewed  cameras  the  1207-PL  is  recommended. 


1.5  Summary  of  the  Other  Products 

The  W5402  lensed  module  is  a  new  kind  of  camera  based  on  CMOS  technology.  The  image 
sensor  is  a  CMOS  array  that  is  integrated  with  the  rest  of  the  electronics  on  a  single  chip.  The 
ability  to  produce  single  chip  cameras  is  a  breakthrough,  and  in  the  long  run  will  allow  the 
production  of  cheaper  high  quality  miniature  video  cameras  in  the  $10-$20  range  as  opposed 
to  the  $100-$200  range  for  CCD  cameras.  Its  overall  score  was  868.  The  camera  has  many 
pluses  -  its  small,  light,  inexpensive,  and  draws  little  power.  There  is  also  the  potential  for  a 
direct  digital  signal.  The  CMOS  camera  we  examined  actually  converts  its  digital  array  to  an 
analog  NTSC  signal.  Our  framegrabber  then  redigitizes  the  signal  for  display  on  a  computer 
monitor.  There  is  the  possibility  of  connecting  a  digital  output  from  a  CMOS  camera  to  a 
universal  serial  bus  (USB)  to  avoid  the  digital  (CMOS)  to  analog  (NTSC)  conversion  followed 
by  the  analog  (NTSC)  to  digital  conversion.  This  could  greatly  improve  the  image  quality  of 
CMOS  cameras.  Although  the  W5402  was  the  second  highest  rated  camera,  it  still  cannot  be 
recommended  for  use  yet.  Its  resolution  and  image  quality  are  not  quite  high  enough  for 
practical  use.  The  VI 247  is  a  1/5  CCD  color  camera  and  was  the  smallest  color  CCD  camera 
we  could  find  at  the  time  of  the  downselect.  Even  though  the  V1247  is  smaller  and  lighter  than 
other  color  cameras,  it  is  still  substantially  heavier  and  larger  than  the  black  and  white  cameras 
in  the  downselect.  Unfortunately,  the  reduction  to  1/5  CCD  to  reduce  the  size  and  power 
consumption  of  the  camera  also  reduced  the  resolution  and  image  quality.  In  addition,  color 
cameras  in  general  require  greater  minimum  illumination  than  to  black  and  white  cameras  (e.g. 
the  1247  was  15  Lux,  while  the  1207  and  1207-PL  were  both  .4  Lux).  This  lack  of  adjustment 
under  low  light  conditions  is  readily  apparent  in  the  jpegs  presented  in  Supplement  A-8-1  -  the 
image  is  much  darker  than  the  other  low  light  images.  Its  overall  score  was  856.  The  VI 207  is 
similar  to  the  1207-PL  except  has  a  larger  form  factor  and  substantially  worse  image  quality. 
Its  overall  score  was  808. 


1.6  Future  Considerations 

When  we  began  this  trade  study,  the  initial  prototypes  to  be  developed  in  the  ITI-ALC  project 
included  live  video  for  collaboration  and  help  desk  scenarios.  In  the  past  month  however,  after 
visiting  RAFB,  we  discovered  that  to  collaborate  with  engineers,  mechanics  might  only  need 
high  resolution,  still,  color  images.  This  is  reflected  in  the  latest  version  of  the  collaboration 
scenario.  While  we  may  move  to  live  video-using  cameras  mounted  on  head  worn  displays  in 
future  scenarios,  it  makes  more  sense  now  to  use  higher  quality  but  bulkier  hand  held  cameras 
for  the  initial  collaboration  scenario.  Digital  image  capturing  devices  that  connect  directly  to 
mobile  computers  may  be  more  appropriate  including  color  cameras  by  Pixera  and  VLSI  Vision 
Limited  (WL). 

As  mentioned  above,  CMOS  cameras  appear  to  be  the  wave  of  the  future  for  although  the 
technology  is  not  quite  mature  for  miniature  video  cameras  or  our  use.  This  should  be 
reevaluated  in  the  next  six  months. 


224 


2.0  Test  Data 


2.1  Resolution 

Using  the  experimental  apparatus  described  in  section  1,  we  collected  still  images  from  each  of 
the  cameras  of  a  test  chart  used  for  assessing  resolution.  We  took  two  images  from  each,  one 
under  high  light  conditions  (with  the  lights  on  in  the  lab),  and  one  with  the  lights  off,  but 
allowing  a  small  amount  of  light  in  through  the  window  blinds.  These  were  meant  to 
correspond  roughly  to  lighting  conditions  in  the  hangar  (1)  near  the  exit  on  a  sunny  day  and  (2) 
inside  a  plane  with  ambient  light.  For  reference,  we  include  the  high  light  and  low  light  images 
for  each  camera  in  Supplement  A-8-1.  Note  that  these  are  scaled  down  to  fit  into  this 
document,  and  so  do  not  represent  the  raw  image  as  viewed  in  the  test  (these  are  much 
smaller  and  so  cannot  be  used  to  assess  resolution).  Still,  they  give  a  relative  comparison  of 
the  different  cameras  and  different  conditions. 

The  test  chart  automatically  normalizes  for  different  cameras  and  lenses.  The  instructions  say 
to  move  the  camera  until  the  full  screen  of  the  television  or  computer  screen  is  taken  up  by 
chart.  Then,  to  get  a  measure  of  the  cameras  resolution,  find  where  the  converging  lines  in  the 
center  and  sides  of  the  charge  are  not  visually  differentiable,  and  use  the  gauge  next  to  the 
lines  to  determine  the  resolution.  We  used  the  chart  in  two  ways  -  the  first  as  described  above 
to  get  an  estimate  of  the  resolution  under  different  lighting  conditions  for  each  camera,  and 
also  to  compare  the  overall  visual  quality  of  the  cameras  by  evaluating  the  captured  images  in 
side  by  side  comparisons  as  described  in  the  next  section.  By  viewing  the  images  on  a 
standard  monitor,  one  person  judged  when  the  lines  became  undifferentiable  for  each  camera 
under  each  condition  (high  light,  low  light).  We  decided  this  method  yielded  a  high  enough 
degree  of  accuracy  for  the  purposes  of  the  trade  study  (a  second  person  double  checked  a  few 
of  the  judgments  with  similar  results).  The  observed  resolution  for  each  of  the  cameras  was: 


Camera 

High  Light  Resolution 

Low  Light  Resolution 

V1207-PL 

425 

350 

VI 247 

350 

300 

VV5402 

325 

310 

VI 207 

275 

240 

Overall,  the  V1207-PL  had  the  highest  observed  resolution  under  both  lighting  conditions.  The 
VI 247  and  the  W5402  we  both  lower  and  had  nearly  the  same  observed  resolution. 
Surprisingly,  the  VI 207  had  the  lowest  observed  resolution,  probably  indicating  that  the 
pinhole  lens  creates  a  sharper  image  than  the  supplied  lens  with  the  VI 207. 

In  all  cases,  the  observed  resolution  for  the  high  light  condition  was  higher  than  the 
corresponding  observed  resolution  in  the  low  light  condition.  This  means  that  regardless  of  the 
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camera  we  choose,  we  must  understand  the  lighting  conditions  in  which  the  camera  will  be 
used,  and  take  that  into  account  in  the  design. 

Note  that  some  of  these  observed  resolutions  differed  from  those  reported  by  the  companies. 
The  V1247  and  W5402  did  much  better  than  their  reported  220  TV  lines,  while  the  V1207-PL 
was  consistent  with  its  reported  380  TV  lines.  The  observed  resolutions  for  VI 207  were  lower 
than  the  value  reported  by  Marshall.  This  variance  is  consistent  with  comments  made  by 
several  independent  vendors  and  experts  we  spoke  with  at  CMU  that  for  resolution,  company 
reported  values  often  differ  considerably  from  those  found  in  actual  use. 


2.2  Assessed  Quality 

In  addition  to  resolution  we  wanted  to  get  an  overall  sense  of  the  still  picture  quality.  We  had 
five  individuals  rank  by  quality  the  images  from  the  high  light  condition  and  the  low  light 
conditions.  The  following  tables  summarize  the  ranking  of  the  images  by  all  individuals: 


Camera/Rank 

1st 

2nd 

3rd 

4th 

V1207-PL 

5 

0 

0 

0 

VI 247 

0 

4 

1 

0 

VI 207 

0 

1 

3 

1 

VV5402 

0 

0 

1 

4 

Camera  Rankings  -  High  Light 


Camera/Rank 

1st 

2nd 

3rd 

4th 

V1207-PL 

5 

0 

0 

0 

VI 247 

0 

1 

3 

1 

VI 207 

0 

1 

1 

3 

W5402 

0 

3 

1 

1 

Camera  Rankings  -  Low  Light 
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Each  cell  in  the  table  gives  the  number  of  individuals  who  gave  the  specified  camera  the 
specified  rank.  By  multiplying  teach  frequency  by  four  minus  the  corresponding  rank  and 
summing  across  the  ranks  for  each  camera  and  across  conditions,  we  get  an  overall 
preference  score  for  each  camera.  The  following  table  presents  the  overall  ranked  scores: 


Camera 

Overall  Preference  Score 

V1207-PL 

40 

VI 247 

24 

VV5402 

18 

VI 207 

18 

Overall  Quality  Score 

In  both  lighting  conditions,  the  V1207-PL  was  ranked  first  by  all  participants,  thus  it  was 
overwhelmingly  preferred  with  a  combined  weighted  score  of  40.  The  VI 247  was  a  distant 
second  followed  by  the  W5402  and  the  VI 207. 
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3.0  Assessment  Tables 


Table  A-8-1  Video  Camera  Trade  Study  -  Product:  Marshall  V1207-PL 


|  Criteria 

Rqmts 

1.0  Performance 

Weight  -  Anything  more  than  2  ounces 
will  be  too  heavy  for  head  worn  use. 
Lighter  is  better.  For  handheld  use,  this  is 
less  of  an  issue. 

Derived 

311005 

Shape/Size  -  The  camera  needs  to  be 
small  enough  so  a  technician  will  not 
accidentally  hit  into  anything  while 
wearing  the  camera.  1.25  inch  cube  is 
upper  bound,  but  flatter  is  better 

Derived 

311005 

Mountability  -  Can  the  camera  be 
mounted  on  a  head  worn  display?  Will 
the  mounting  be  stable  and  rugged? 

Derived 

311005 

Power  -  2W  is  rough  upper  bound 
depending  on  battery  technology  and  if 
continuous  or  sporadic  use. 

Derived 

311013 

1.1  Performance  -  assessed  picture  quality 

Resolution  -  High  enough  to  display  live 
video  and  snapshots  to  enable 
collaboration.  Minimum  resolution  is 
probably  220  Horizontal  TV  lines. 

Higher  (300-500  range)  is  desirable. 
Actual  display  resolution  depends  on 
bandwidth  and  software  drivers. 

Derived 

311007 

Quality  (Still  Print) 

Derived 

311007 

Color  or  Black  and  White  Video-  There 
is  a  trade-off  between  size  and  picture 
quality.  (Color  cameras  are  larger)  We 
still  need  to  determine  how  useful  color 
will  be. 

Derived 

311005 

Minimum  Illumination  -  The  camera 
needs  to  be  able  to  operate  in  the  range  of 
lighting  conditions  that  occur  at  the 
depot.  1 5  lux  is  upper  bound  for 
minimum. 

Derived 

311005 

Iris  Control  -  Does  camera  have 
automatic  (electronic)  control?  This  helps 
in  adjusting  to  different  lighting 
conditions 

Derived 

311005 

Comments 


.7  Oz  (20grams) 


1.5”  XI. 5”  X  .57”  D 


mount  will  need  to  be 
constructed,  but  should  not  be 
hard 


Less  power  than  the  VI 247, 
same  power  VI 207,  more 
power  than  the  VV5402. 


537Hx505V  (380  TV  lines) 
reported 

425/350  (High/Low)  observed 
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Table  A-8-1  Video  Camera  Trade  Study  -  Product:  Marshall  V1207-PL  (Continued) 


Lens  Mount  -  Can  a  variety  of  lenses  be 
attached?  We  need  to  be  able  to  hook  up 
lenses  with  a  range  of  Field  of  View  (35 
to  90  degrees) 

Derived 

311005 

9 

8 

72 

Yes,  wide,  clear  FOV  (74 
degrees  horizontal,  by  56 
degrees  vertical)  and  wide 
angle  lens  available 

White  Balance  -  (color  cameras  only)  - 
Should  be  automatic  -  another  factor  in 
image  quality 

Derived 

311005 

7 

NA 

na 

2.0  Reliability 


Ruggedness  -  Camera  needs  to  be  able  to 
take  impact  and  keep  working  if  someone 
accidentally  knocks  it.  i.e.,  needs  strong 
housing  and  stable  camera  mount. 
(Vibration  and  shock  values  of  7G  and 
70G  are  excellent). 

Derived 

311005 

8 

7 

56 

More  rugged  than  the  VI 247, 
at  least  as  rugged  as  the  other 
cameras  (no  hard  data 
obtainable) 

Operating  Temperature  -  Camera 
should  be  able  to  work  in  wide  range  of 
temperatures  (-10  C  to  50  C) 

Derived 

311017 

7 

10 

70 

-10C  -  +55C 

3.0  Maintainability 


Not  Applicable 

_ 

4.0  Producibility 


Cost  -  most  of  these  cameras  range  from 
$1004250. 

6 

$155 

5.0  Capability 

Output  Signal  -  The  camera  should 
transmit  output  in  an  appropriate  format 
for  the  wearable  computer  (NTSC, 
digital,  etc.). 

9 

10 

90 

NTSC 

6.0  Dependencies 

Bandwidth  -  for  use  in  collaboration,  the 
bandwidth  between  computers  will  affect 
the  quality  of  the  video  signal.  E.g.  for 
continuous  video,  if  bandwidth  is  low, 
increasing  the  camera  resolution  will  not 
improve  the  video  signal 

Derived 

311002 

8 

Table  A-8-2  Video  Camera:  WL  5402 


Criteria 


1.0  Performance 


Rqmts  Weight  Score  Total  Comments 


Weight  -  Anything  more  than  2  ounces 
will  be  too  heavy  for  head  worn  use. 
Lighter  is  better.  For  handheld  use,  this  is 
less  of  an  issue. 


Shape/Size  -  The  camera  needs  to  be 
small  enough  so  a  technician  will  not 
accidentally  hit  into  anything  while 
wearing  the  camera.  1.25  inch  cube  is 
upper  bound,  but  flatter  is  better 

Derived 

311005 

Mountability  -  Can  the  camera  be 
mounted  on  a  head  worn  display?  Will 
the  mounting  be  stable  and  rugged? 

Derived 

311005 

Power  -  2W  is  rough  upper  bound 
depending  on  battery  technology  and  if 
continuous  or  sporadic  use. 

Derived 

311013 

1.1  Performance  -  assessed  picture  quality 

Resolution  -  High  enough  to  display  live 
video  and  snapshots  to  enable 
collaboration.  Minimum  resolution  is 
probably  220  Horizontal  TV  lines. 

Higher  (300-500  range)  is  desirable. 
Actual  display  resolution  depends  on 
bandwidth  and  software  drivers. 

Derived 

311007 

Quality  (Still  Print) 

Derived 

311007 

Color  or  Black  and  White  Video-  There 
is  a  trade-off  between  size  and  picture 
quality.  (Color  cameras  are  larger)  We 
still  need  to  determine  how  useful  color 
will  be. 

Derived 

311005 

Minimum  Illumination  -  The  camera 
needs  to  be  able  to  operate  in  the  range  of 
lighting  conditions  that  occur  at  the 
depot.  15  lux  is  upper  bound  for 
minimum. 

Derived 

311005 

Iris  Control  -  Does  camera  have 
automatic  (electronic)  control?  This  helps 
in  adjusting  to  different  lighting 
conditions 

Derived 

311005 

22mm(h)x22mm(w)x24mm(d) 


7  63  Need  to  construct  mount  (yes) 


384x287 

(265  TVL  reported) 

325/310  (High/low)  observed 
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Table  A-8-2  Video  Camera:  VVL  5402 


Lens  Mount  -  Can  a  variety  of  lenses  be  Derived  9  8  72 

attached?  We  need  to  be  able  to  hook  up  311005 
lenses  with  a  range  of  Field  of  View  (35 
to  90  degrees) 

White  Balance  -  (color  cameras  only)  -  Derived  7  na 

Should  be  automatic  -  another  factor  in  31 1005 
image  quality 


2.0  Reliability 


Ruggedness  -  Camera  needs  to  be  able  to 
take  impact  and  keep  working  if  someone 
accidentally  knocks  it.  i.e.,  needs  strong 
housing  and  stable  camera  mount. 
(Vibration  and  shock  values  of  7G  and 
70G  are  excellent). 

Derived 

311005 

8 

7 

56 

More  rugged  than  the  VI 247, 
as  rugged  as  the  other  cameras 
(No  hard  data  obtainable) 

Operating  Temperature  -  Camera 
should  be  able  to  work  in  wide  range  of 
temperatures  (-10  C  to  50  C) 

Derived 

311017 

7 

6 

42 

0c  -  40c 

3.0  Maintainability 


Not  Applicable 
4.0  Producibility 


Cost  -  most  of  these  cameras  range  from 

6 

10 

60 

$100  now  with  potential  to 

$100-$250. 

drop  to  $10  in  bulk  purchases 

5.0  Capability 

Output  Signal  -  The  camera  should 
transmit  output  in  an  appropriate  format 
for  the  wearable  computer  (NTSC, 
digital,  etc.). _ 

6.0  Dependencies 

Bandwidth  -  for  use  in  collaboration,  the  Derived  8 
bandwidth  between  computers  will  affect  311002 
the  quality  of  the  video  signal.  E.g.  for 
continuous  video,  if  bandwidth  is  low, 
increasing  the  camera  resolution  will  not 

improve  the  video  signal _ 

I  Total:  110  \~ 868 


Derived 

9 

10 

90 

EIA  (NTSC),  possibility  of 

311002 

digital 

Yes 


NA 
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Table  A-8-3  Video  Camera  Trade  Study  -  Product:  Marshall  VI 247 


Criteria 


1.0  Performance 


Rqmts I  Weight  I  Score  Total  Comments 


Weight  -  Anything  more  than  2  ounces 
will  be  too  heavy  for  head  worn  use. 
Lighter  is  better.  For  handheld  use,  this  is 
less  of  an  issue. 

Derived 

311005 

Shape/Size  -  The  camera  needs  to  be 
small  enough  so  a  technician  will  not 
accidentally  hit  into  anything  while 
wearing  the  camera.  1 .25  inch  cube  is 
upper  bound,  but  flatter  is  better 

Derived 

311005 

Mountability  -  Can  the  camera  be 
mounted  on  a  head  worn  display?  Will 
the  mounting  be  stable  and  rugged? 

Derived 

311005 

Power  -  2W  is  rough  upper  bound 
depending  on  battery  technology  and  if 
continuous  or  sporadic  use. 

Derived 

311013 

1.1  Performance  -  assessed  picture  quality 

Resolution  -  High  enough  to  display  live 
video  and  snapshots  to  enable 
collaboration.  Minimum  resolution  is 
probably  220  Horizontal  TV  lines. 

Higher  (300-500  range)  is  desirable. 
Actual  display  resolution  depends  on 
bandwidth  and  software  drivers. 

Derived 

311007 

Quality  (Still  Print) 

Derived 

311007 

Color  or  Black  and  White  Video-  There 
is  a  trade-off  between  size  and  picture 
quality.  (Color  cameras  are  larger)  We 
still  need  to  determine  how  useful  color 
will  be. 

Derived 

311005 

Minimum  Illumination  -  The  camera 
needs  to  be  able  to  operate  in  the  range  of 
lighting  conditions  that  occur  at  the 
depot.  15  lux  is  upper  bound  for 
minimum. 

Derived 

311005 

Iris  Control  -  Does  camera  have 
automatic  (electronic)  control?  This  helps 
in  adjusting  to  different  lighting 
conditions 

Derived 

311005 

5 

45 

1 .4  Oz  (40  grams) 

5 

45 

1.5”Lxl.5”Wxl.75”H 

7 

63 

mount  will  need  to  be 
constructed,  but  should  not  be 
hard 

6 

42 

2.16W 

7 

63 

(220  TV  lines)  reported 

350/300  (High/Low)  observed 

6 

54 

overall  quality:  24 

10 

50 

color 

5 

40 

15  Lux 

10 

70 

yes 
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Table  A-8-3  Video  Camera  Trade  Study  -  Product:  Marshall  V1247  (Continued) 


Lens  Mount  -  Can  a  variety  of  lenses  be 
attached?  We  need  to  be  able  to  hook  up 
lenses  with  a  range  of  Field  of  View  (35 
to  90  degrees) 

Derived 

311005 

9 

8 

72 

yes 

White  Balance  -  (color  cameras  only)  - 
Should  be  automatic  -  another  factor  in 
image  quality 

Derived 

311005 

7 

10 

70 

yes 

2.0  Reliability 

Ruggedness  -  Camera  needs  to  be  able  to 
take  impact  and  keep  working  if  someone 
accidentally  knocks  it.  I.e.  needs  strong 
housing  and  stable  camera  mount. 
(Vibration  and  shock  values  of  7G  and 

70G  are  excellent). 

Derived 

311005 

8 

5 

40 

Least  rugged  of  the  four 
cameras  due  to  its  two 
connected  boards  (no  hard  data 
obtainable) 

Operating  Temperature  -  Camera 
should  be  able  to  work  in  wide  range  of 
temperatures  (-10  C  to  50  C) 

Derived 

311017 

7 

10 

70 

-10C-+55C 

-20  -  +70C  in  storage 

3.0  Maintainability 

Not  Applicable 
4.0  Producibility 


Cost  -  most  of  these  cameras  range  from  6  7  42  $245 

$100-$250. _ _ _ _ _ 

5.0  Capability 

Output  Signal  -  The  camera  should  Derived  9  10  90  NTSC 

transmit  output  in  an  appropriate  format  311002 
for  the  wearable  computer  (NTSC, 

digital,  etc.). _ 

6.0  Dependencies 

Bandwidth  -  for  use  in  collaboration,  the  Derived  8 
bandwidth  between  computers  will  affect  311002 
the  quality  of  the  video  signal.  E.g.  for 
continuous  video,  if  bandwidth  is  low, 
increasing  the  camera  resolution  will  not 

improve  the  video  signal _ 

I  Total:  I  111  856  1 
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Table  A-8-4  Video  Camera  Trade  Study  -  Product:  Marshall  VI 207 


Criteria 


1.0  Performance 


Rqmts  Weight  Score  Total  Comments 


Weight  -  Anything  more  than  2  ounces 
will  be  too  heavy  for  head  worn  use. 
Lighter  is  better.  For  handheld  use,  this  is 
less  of  an  issue.  ___ 


Shape/Size  -  The  camera  needs  to  be 
small  enough  so  a  technician  will  not 
accidentally  hit  into  anything  while 
wearing  the  camera.  1 .25  inch  cube  is 
upper  bound,  but  flatter  is  better _ 


Mountability  -  Can  the  camera  be 
mounted  on  a  head  worn  display?  Will 
the  mounting  be  stable  and  rugged? 


Power  -  2W  is  rough  upper  bound 
depending  on  battery  technology  and  if 
continuous  or  sporadic  use. 


1.1  Performance  -  assessed  picture  quality 


Derived 

311013 


Resolution  -  High  enough  to  display  live 
video  and  snapshots  to  enable 
collaboration.  Minimum  resolution  is 
probably  220  Horizontal  TV  lines. 
Higher  (300-500  range)  is  desirable. 
Actual  display  resolution  depends  on 
bandwidth  and  software  drivers. 


Quality  (still  print) 


Derived 

311007 


Derived 

311007 


Color  or  Black  and  White  Video  - 

There  is  a  trade-off  between  size  and 
picture  quality.  (Color  cameras  are 
larger)  We  still  need  to  determine  how 
useful  color  will  be. 


Minimum  Illumination  -  The  camera  Derived 
needs  to  be  able  to  operate  in  the  range  of  311005 
lighting  conditions  that  occur  at  the 
depot.  15  lux  is  upper  bound  for 
minimum.  _ 


Iris  Control  -  Does  camera  have  Derived 

automatic  (electronic)  control?  This  helps  311005 
in  adjusting  to  different  lighting 
conditions 


72 

.7  Oz  (20grams) 

| 

63 

1 .25”  X 1 .25”  X  .  1 .25”  D 

63 

mount  will  need  to  be 
constructed,  but  should  not  be 
hard 

49 

Lower  than  the  VI 247,  higher 
than  the  VV5402 

32 

537Hx505V  (380  TV  lines) 
reported 

275/240  (hi/low) 
observed 

Overall  Quality:  1 8 

25 

b&w 

72 

.4  Lux 

70 

yes 

234 


Table  A-8-4  Video  Camera  Trade  Study  -  Product:  Marshall  V1207 


Lens  Mount  -  Can  a  variety  of  lenses  be  Derived 
attached?  We  need  to  be  able  to  hook  up  311005 
lenses  with  a  range  of  Field  of  View  (35 
to  90  degrees) 


White  Balance  -  (color  cameras  only)  -  Derived 
Should  be  automatic  -  another  factor  in  311005 
image  quality 


2.0  Reliability 


Ruggedness  -  Camera  needs  to  be  able  to  Derived 
take  impact  and  keep  working  if  someone  311005 
accidentally  knocks  it.  i.e.,  needs  strong 
housing  and  stable  camera  mount. 

(Vibration  and  shock  values  of  7G  and 

70 G  are  excellent). _ 

Operating  Temperature  -  Camera  Derived 

should  be  able  to  work  in  wide  range  of  311017 
temperatures  (-10  C  to  50  C) 


3.0  Maintainability 


More  rugged  than  the  VI 247, 
probably  as  rugged  as  the 
other  cameras  (no  hard  data 
obtainable) 


-10C  -  +55C 


Not  Applicable 
4.0  Producibility 

Cost  -  most  of  these  cameras  range  from 
$100-$250. 


5.0  Capability 


Output  Signal  -  The  camera  should 
transmit  output  in  an  appropriate  format 
for  the  wearable  computer  (NTSC, 
digital,  etc.). 


6.0  Dependencies 


Derived 

311002 


Bandwidth  -  for  use  in  collaboration,  the  Derived 
bandwidth  between  computers  will  affect  311002 
the  quality  of  the  video  signal.  E.g.  for 
continuous  video,  if  bandwidth  is  low, 
increasing  the  camera  resolution  will  not 
improve  the  video  signal 
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4.0  Experience 

In  addition  to  performing  the  tests  described  in  section  two,  we  informally  evaluated  the 
cameras  using  a  direct  hook  up  to  a  TV  monitor  and  viewing  people,  objects,  and  written  text. 
By  far,  the  V1207-PL  had  the  clearest  image,  with  the  VI 247  and  VI 207  close  behind.  The 
color  in  the  VI 247  gave  a  greater  sense  of  reality  to  the  video  and  allowed  a  greater  distinction 
between  objects,  when  the  room  had  sufficient  light  for  the  color  contrasts  to  come  through. 
The  video  from  W5402  was  the  worst,  regardless  of  the  lenses  we  tried  with  it.  The  contrast  in 
the  W5402  seemed  lower  than  that  of  the  other  cameras,  resulting  in  a  smoother  but  more 
washed  out  video  image. 


5.0  Product  Data  Sheets 

Product  data  sheets  can  be  obtained  from  the  following  web  sites: 

Marshall  Cameras:  http://www.mars-cam.com/ 

VLSI  Vision  Ltd.:  http://www.vvl.co.uk/ 
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APPENDIX  B 

USABILITY  EVENTS  QUESTIONNAIRES 
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Appendix  B:  Usability  Events  Questionnaires 

ITI-ALC  Phase  II 
FBE#2 

Post  Questionnaire 


Name _ 

1 .  Did  you  use  the  mobile  prototype? 
_ Yes  _ No 

If  "No",  why  not? 


Please  go  to  question  8. 

If  yes,  please  answer  questions  2  through  10. 


Please^  circle  your  answer 


Very  Easy 

Very  Difficult 

2. 

Seeing  the  text  in  the  application  was 

1 

2 

3 

4 

5 

3. 

Seeing  the  graphic  representation  of  the  aircraft 
was 

1 

2 

:> 

4 

5 

4. 

Finding  the  information  I  needed  for  the  E&I 
inspection  or  Form  202  task  was 

1 

2 

3 

4 

5 

5.  What  do  you  like  best  about  using  the  mobile  system  to  access  the  F-l  5  E  &I  or  Form  202 
information? 


6.  What  do  you  like  least  about  using  the  mobile  to  access  the  F-l 5  E  &I  or  Form  202  information? 
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7.  For  the  F- 15  E&  I  inspection  or  the  Form  202  task,  was  there  any  information  you  needed  that  was 
unavailable  to  you?  _ _ yes _ no 

If  “yes”,  please  describe  briefly  _ 


8.  What  would  you  change  about  the  mobile  system  if  you  could?  (for  example,  would  you  want:  bigger 
screen,  faster  response  time,  fewer  handwriting  recognition  errors,  easier  ways  to  change  responses) 


9.  Please  describe  briefly  how  you  felt  about  using  the  computer  to  record  information  instead  of  using 
paper: 


10.  Is  there  anything  else  you  would  like  us  to  know  about  your  experience  with  mobile  computing 
today? 


Thank  you  for  your  participation  in  this  evaluation  session. 
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APPENDIX  C 

FIELD  BASED  EVALUATION  QUESTIONS 
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Appendix  C:  Field  Based  Evaluation  Questions 


The  following  questions  are  about  the  ITI-ALC  system  you  just  used.  The  questions  are  grouped 
in  three  sections.  The  first  section  asks  general  questions  about  this  kind  of  maintenance  aid.  In  other 
words,  not  necessarily  this  system,  but  one  like  it.  The  second  section  asks  questions  about  how  the  ITI- 
ALC  system  effects  your  ability  to  perform  your  job.  The  third  section  asks  questions  about  how  easy 
the  hardware  and  software  are  to  use. 

Please  answer  all  questions  that  you  can.  Indicate  your  answer  by  circling  the  choice  you  select. 
If  you  cannot  answer  a  question,  leave  it  blank. 

ITI-ALC  CONCEPT 

1 .  Would  an  ITI-ALC  system  increase  your  use  of  TO  data? 

[Al]  Not  at  all  1  2  3  4  5  Definitely 

2.  Would  an  ITI-ALC  system  increase  access  to  most  types  of  information  needed  for  your  job? 

[A2]Yes  No 
If  no,  why  not 


3.  An  ITI-ALC  system  would  make  estimates  of  task  time  to  complete  [A3] 

More  accurate  1  2  3  4  5  Less  accurate 

4.  If  an  ITI-ALC  system  were  available,  would  you  draw  with  it?[A4] 

Yes  No  Maybe 

5.  Would  an  ITI-ALC  system  be  easy  for  a  novice  maintenance  technician  to  use?  [A5] 

Yes  No  Maybe 
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6.  Performing  tasks  outside  your  specialty  with  an  ITI-ALC  system  would  be[A6] 

Possible  1  2  3  4  5  Impossible 

7.  If  you  had  an  ITI-ALC  system,  would  you  use  it  for  your  primary  job?  [ A7] 

Yes  No  Depends 

If  no,  why  not 


8.  If  an  ITI-ALC  system  were  available  to  you,  you  would  use  it[A8] 

Never  1  2  3  4  5  Frequently 

9.  Are  there  maintenance  areas  where  an  ITI-ALC  system  would  be  useful? 

Yes  No 

If  yes,  which  ones?  [A9] 


10.  Are  there  maintenance  areas  where  an  ITI-ALC  system  would  NOT  be  useful? 
Yes  No 

If  yes,  which  ones?  [A1 0] 
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ITI-ALC’s  EFFECT  ON  YOUR  JOB  PERFORMANCE 


1 .  Obtaining  information  using  the  ITI-ALC  system  as  compared  to  current  TOs  was  (Answer  both  sets 
of  responses) 

[All]  Faster  Slower  Same 

Easier  Harder  Same 

2.  Obtaining  forms  using  the  ITI-ALC  system  as  compared  to  the  current  process  for  obtaining  forms 
was  (Answer  both  sets  of  responses) 

[A12]  Faster  Slower  Same 

Easier  Harder  Same 

3.  Ability  to  access  other  types  of  information  (e.g.,  parts  information)  using  the  ITI-ALC  system  as 
compared  to  the  current  method  was  (Answer  both  sets  of  responses) 


Faster 

Slower 

Same 

Easier 

Harder 

Same 

4.  Compared  to  the  current  process,  ordering  parts  with  ITI-ALC  would  be  was  (Answer  both  sets  of 
responses) 

[A13]  Faster  Slower  About  the  same 
Easier  Harder  Same 

5.  Time  required  to  enter  information  into  the  ITI-ALC  system  as  opposed  to  the  current  method  was 
[A  14] 

Longer  Shorter  Same 
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6.  To  accomplish  most  tasks,  the  digital  camera  provided[Al  5] 

Inadequate  detail  1  2  3  4  5  Sufficient  detail 

7.  If  you  had  digital  photos  of  damaged  areas,  the  number  of  trips  to  the  aircraft  to  view  the  damage 
would  [A16] 

Remain  the  same  Decrease 

8.  Use  of  the  I-button  would  make  the  signature  approval  time[Al  7] 

Faster  1  2  3  4  5  Slower 

9.  In  comparison  with  the  current  signature  security  process,  the  I-button  would  make  signature 
approval  security  [A18] 

Less  secure  1  2  3  4  5  More  secure 

10.  Compared  to  the  current  system,  time  required  to  send  information  using  the  ITI-ALC  system  was 
[A19] 

Longer  Shorter  Same 

11.  While  carrying/wearing  the  system,  it  interfered  with  the  maintenance  task  [A20] 

Infrequently  1  2  3  4  5  Frequently 

12.  As  compared  with  paper,  information  presented  on  the  ITI-ALC  system  was  [A21] 

Easier  to  read  Harder  to  read  About  the  same 


ITI-ALC  HARDWARE  AND  SOFTWARE  USE 

1 .  Carrying/wearing  the  ITI-ALC  system  while  performing  your  task  was  [ A22] 

Convenient  1  2  3  4  5  Inconvenient 

2.  Was  the  system  comfortable  to  wear?  [A23] 

Yes  No 

3.  Did  the  straps  fasten  the  ITI-ALC  system  securely  to  your  body?  [A24] 

Yes  No 
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4.  While  using  the  computer  to  perform  your  task,  the  use  of  the  pen  was: 

Easy  1  2  3  4  5  Difficult 

5.  Was  it  obvious  when  the  computer  was  busy  working?  [A25] 

Yes  No 

If  no,  when 


6.  Information  presented  on  the  ITI-ALC  system  was  [A26] 

Easy  to  read  1  2  3  4  5  Difficult  to  read 

7.  Information  you  entered  into  the  ITI-ALC  system  was[A27] 

Easy  to  read  1  2  3  4  5  Difficult  to  read 

8.  Information  presented  on  the  ITI-ALC  system  was  [A28] 

Adequate  1  2  3  4  5  Inadequate 

9.  Entering  text  using  the  on-screen  keyboard  was  [A29] 

Easy  1  2  3  4  5  Difficult 

10.  The  meaning  of  the  “stamp”  icon  was 

Not  obvious  Obvious 

If  it  was  not  obvious,  what  would  you  suggest? 
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1 1 .  The  meaning  of  the  “envelope”  icon 


Not  obvious  Obvious 

If  it  was  not  obvious,  what  would  you  suggest? 


12.  The  navigation  bar  on  the  right  side  of  the  computer  was 

Easy  to  understand  1  2  3  4  5  Difficult  to  understand 

13.  Entering  sketches  on  the  ITI-ALC  system  was[A30] 

Easy  1  2  3  4  5  Difficult 

14.  Using  the  camera  to  photograph  damaged  area  was  [A31] 

Easy  1  2  3  4  5  Difficult 

15.  Reviewing  photos  you  had  taken  was[A32] 

Easy  1  2  3  4  5  Difficult 

16.  Did  you  notice  when  photos  were  available  for  you  to  review[A33] 

Yes  No 

17.  Saving  photos  on  the  computer  was[A34] 

Easy  1  2  3  4  5  Difficult 

18.  Viewing  photos  was  [A35] 

Easy  1  2  3  4  5  Difficult 

19.  Recording  voice  notes  was[A36] 

Easy  1  2  3  4  5  Difficult 
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20.  Playing  back  voice  notes  was[A37] 

Easy  1  2  3  4  5  Difficult 

21.  Reviewing  voice  notes  you  had  recorded  was  [A38] 

Easy  1  2  3  4  5  Difficult 

22.  Did  you  notice  when  voice  notes  were  available  for  you  to  review  [A39] 

Yes  No 

23.  Transmitting  information  for  other  people  to  read  or  approve  was  [A40] 

Easy  1  2  3  4  5  Difficult 

24.  Did  you  know  when  the  computer  was  sending  data?  [A41] 

Yes  No 

25.  The  ITI-ALC  system  was  [A42] 

Not  useful  1  2  3  4  5  Very  Useful 

26.  What  did  you  like  most  about  the  ITI-ALC  system?[A43] 


27.  What  did  you  like  least  about  the  ITI-ALC  system?[A44] 
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Did  you  use  an  ITI-  ALC  system  in  a  previous  study? 

Yes  No  Don’t  remember 

28.  If  yes,  which  one  (screen  presentations  primarily)  did  you  prefer?  [A45] 
Previous  ITI- ALC  system  This  ITI- ALC  system 
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APPENDIX  D 

COMPLETED  SHAKEDOWN  EVENT  QUESTIONNAIRES 
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Appendix  D:  Completed  Shakedown  Event  Questionnaires 

1.0  AFRL  Participant  1 

1 .  Did  you  use  the  mobile  prototype?  X  Yes  _  No 

If  "No",  why  not? 


Please  go  to  question  8. 

If  yes,  please  answer  questions  2  through  8. 


Please  circle^  your  answer^ 


Very  Easy 

Very  Difficult 

2. 

Seeing  the  text  in  the  application  was 

1 

X 

2 

3 

4  5 

3. 

Seeing  the  graphic  representation  of  the  aircraft 

1 

2 

3 

4  5 

was 

X 

4. 

Finding  the  information  I  needed  for  the 

1 

2 

o 

J> 

4  5 

inspection  or  Form  202  task  was 

X 

5.  What  do  you  like  best  about  using  the  mobile  system  to  access  the  F-l  5  E  &I  or  Form  202 
information? 

very  simple  to  work  with  and  can  be  used  in  any  environment,  paperwork  tends  to  get  lost,  blown  away 
or  rained  on 


6.  What  do  you  like  least  about  using  the  mobile  to  access  the  F- 1 5  E  &I  or  Form  202  information? 
didn’t  have  anything  I  disliked,  but  just  might  be  easier  to  find  a  blank  defect  form  button  on  the  main 
screen 


7.  For  the  F-l 5  E&I  inspection  or  the  Form  202  task,  was  there  any  information  you  needed  that  was 

unavailable  to  you?  X  yes _ no 

If  yes,  please  describe  briefly 

how  to  determine  exactly  what  T.O.  to  use  when  creating  the  202  from  the  blank  form 
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8.  What  would  you  change  about  the  mobile  system  if  you  could?  (for  example,  would  you  want:  bigger 
screen,  faster  response  time,  fewer  handwriting  recognition  errors,  easier  ways  to  change  responses) 
probably,  mv  biggest  difficulty  was  using  the  pen  and  identifying  characters  from  the  pen,  seem  to  take 
many  tries  to  accomplish.  I  think  also  a  smaller  unit  to  fit  in  one  hand  for  cockpit  areas  and  inspections 
in  tighter  areas 


9.  Please  describe  briefly  how  you  felt  about  using  the  computer  to  record  information  instead  of  using 
paper: 

I  thought  the  computer  was  very  helpful,  and  much  better  than  the  paper  process  we  use  now.  It  could  be 
set  up  even  easier  by  nutting  steps  in  a  logical  sequence  (i.e.  step  1,  step  2,  etc) 


10.  Is  there  anything  else  you  would  like  us  to  know  about  your  experience  with  mobile  computer 
today? 

I  enjoyed  taking  part  of  this  testing  of  the  mobile  system.  I  found  it  very  simple  and  very  useful  and 
hope  to  use  this  system  in  the  future 


2.0  AFRL  Participant  2 

1 .  Did  you  use  the  mobile  prototype?  X  Y es  _  No 

If  "No",  why  not? 


Please  go  to  question  8. 

If  yes,  please  answer  questions  2  through  8. 


Please  circle  yaw  answer: 


Very  Easy 

Very  Difficult 

2. 

Seeing  the  text  in  the  application  was 

1 

X 

2 

3 

4 

5 

3. 

Seeing  the  graphic  representation  of  the  aircraft 
was 

1 

X(3a) 

2 

3 

4 

5 

X(3b) 

4. 

Finding  the  information  I  needed  for  the 

1 

2 

3 

4 

5 

inspection  or  Form  202  task  was 

X 

5.  What  do  you  like  best  about  using  the  mobile  system  to  access  the  F-l  5  E  &I  or  Form  202 
information? 
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It  was  simple  to  use.  It’s  a  quick  wav  to  move  information  to  decrease  time 


6.  What  do  you  like  least  about  using  the  mobile  to  access  the  F-15  E  &I  or  Form  202  information? 
The  pen  was  easy  to  use. . .but  at  times  it  was  sensitive  and  did  not  respond,  (multiple  use  for  a  single 
action) 


7.  For  the  F-15  E&I  inspection  or  the  Form  202  task,  was  there  any  information  you  needed  that  was 

unavailable  to  you?  X  yes _ no 

If  yes,  please  describe  briefly 

Question  “3B”  The  inspection  called  for  information  out  of  IF.15E-3-8,  which  the  3-8  didn’t  have. 


8.  What  would  you  change  about  the  mobile  system  if  you  could?  (for  example,  would  you  want:  bigger 
screen,  faster  response  time,  fewer  handwriting  recognition  errors,  easier  ways  to  change  responses) 

Like  I  mentioned  reduce  the  sensitivity  of  the  pen,  everyone  wants  faster  response  time  from  computers! 
Using  the  pen  and  touching  keyboard  buttons  can  be  tedious  after  a  while 


9.  Please  describe  briefly  how  you  felt  about  using  the  computer  to  record  information  instead  of  using 
paper: 

The  system  seems  to  work  great.  I  have  some  computer  background,  so  it  was  easy  to  pick  up.  My 
only  concern  with  all  computer  programs  is  “If  they  crash  no  wav  around  it.”  Paper  does  fail 


10.  Is  there  anything  else  you  would  like  us  to  know  about  your  experience  with  mobile  computer 
today? 

It  was  fun.  I  enjoyed  it.  a  good  system 


3.0  AFRL  Participant  3 

1 .  Did  you  use  the  mobile  prototype?  X  Y es  _  No 

If  "No",  why  not? 

Please  go  to  question  8. 

If  yes,  please  answer  questions  2  through  8. 
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Please  circle  your  answer^ 


Very  Easy 

Very  Difficult 

2. 

Seeing  the  text  in  the  application  was 

1 

X 

2 

3 

4 

5 

3. 

Seeing  the  graphic  representation  of  the  aircraft 
was 

1 

2 

X 

3 

4 

5 

4. 

Finding  the  information  I  needed  for  the 
inspection  or  Form  202  task  was 

1 

2 

X 

3 

4 

5 

5.  What  do  you  like  best  about  using  the  mobile  system  to  access  the  F-l  5  E  &I  or  Form  202 
information? 

I  liked  the  easy  access  to  the  task  listing  and  ability  to  pull  up  common  defects  directly  from  the  task 
listing,  using  the  icon 


6.  What  do  you  like  least  about  using  the  mobile  to  access  the  F-l 5  E  &I  or  Form  202  information? 

I  didn’t  like  being  sent  back  to  the  ton  of  the  task  list  once  defects  were  noted.  I  would  have  preferred  to 
be  sent  back  to  the  area  on  the  task  listing  that  I  left 


7.  For  the  F-l  5  E&I  inspection  or  the  Form  202  task,  was  there  any  information  you  needed  that  was 
unavailable  to  you? _ yes  X  no 

If  yes,  please  describe  briefly _ N/A _ _ 


8.  What  would  you  change  about  the  mobile  system  if  you  could?  (for  example,  would  you  want:  bigger 
screen,  faster  response  time,  fewer  handwriting  recognition  errors,  easier  ways  to  change  responses) 
Faster  response  times,  better  ability  to  click  (had  problems  with  the  pen  system).  Also  would  find  a  way 
to  let  technician  know  if  they  were  about  to  exit  without  submitting  defects  during  that  session 


9.  Please  describe  briefly  how  you  felt  about  using  the  computer  to  record  information  instead  of  using 
paper: 

Loved  it. 


10.  Is  there  anything  else  you  would  like  us  to  know  about  your  experience  with  mobile  computer 
today? 

I  found  it  very  useful  as  a  guide  of  how  to  perform  E&I  inspection.  For  non-experienced  inspectors,  this 
would  be  a  great  tool 
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4.0  AFRL  Participant  4 

1 .  Did  you  use  the  mobile  prototype?  X  Yes  _  No 

If  "No",  why  not? 


Please  go  to  question  8. 


If  yes,  please  answer  questions  2  through  8. 
Please  circle  your  answer^ _ 


Very  Easy 

Very  Difficult 

2. 

Seeing  the  text  in  the  application  was 

1 

2 

X 

3 

4  5 

3. 

Seeing  the  graphic  representation  of  the  aircraft 

1 

2 

3 

4  5 

was 

X 

4. 

Finding  the  information  I  needed  for  the 

1 

2 

3 

4  5 

inspection  or  Form  202  task  was 

X 

5.  What  do  you  like  best  about  using  the  mobile  system  to  access  the  F-15  E  &I  or  Form  202 
information? 

System  uses  the  same  software  currently  used  throughout  industry 


6.  What  do  you  like  least  about  using  the  mobile  to  access  the  F-l  5  E  &I  or  Form  202  information? 
It  should  be  versatile,  meaning  touch  with  fingers  or  the  pen 


7.  For  the  F-l 5  E&I  inspection  or  the  Form  202  task,  was  there  any  information  you  needed  that  was 
unavailable  to  you? _ yes  X  no 

If  yes,  please  describe  briefly  _ N/A  _ 


8.  What  would  you  change  about  the  mobile  system  if  you  could?  (for  example,  would  you  want:  bigger 
screen,  faster  response  time,  fewer  handwriting  recognition  errors,  easier  ways  to  change  responses) 
Bigger  screen  would  help,  but  if  the  fields  were  a  little  bigger  or  if  we  had  the  option  to  increase  or 
decrease  size  would  help.  Need  feedback  to  mechanic  before  he  sends  the  202A  that  a  voice  and  picture 
are  really  attached.  The  bungee  material  used  on  the  pen  tends  to  wrap  around  the  device. 


256 


9.  Please  describe  briefly  how  you  felt  about  using  the  computer  to  record  information  instead  of  using 
paper: 

Much  easier  and  handier  than  utilizing  3  or  4  items  that  thins  unit  replaces 


10.  Is  there  anything  else  you  would  like  us  to  know  about  your  experience  with  mobile  computer 
today? 
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APPENDIX  E 

QUESTIONNAIRE,  PARTICIPANT  COMMENTS  AND 
TEST  ADMINISTRATOR  OBSERVATIONS 
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Appendix  E: 


Questionnaire,  Participant  Comments  and  Test  Administrator  Observations 

ITI-ALC  Concept 

1 .  Would  an  ITI-ALC  system  increase  your  use  of  TO  data? 

Inspectors 
Not  applicable 
Didn’t  use  TOs 
Mechanics 

If  TO  is  checked  out  then  you  have  to  hunt  for  it 


2.  Would  an  ITI-ALC  system  increase  access  to  most  types  of  information  needed  for  your  job? 
Inspectors 

Don  7  know 

Mechanics 

Availability 

But  the  system  was  very  very  slow  searching  for  information  from  the  TOs 
[Some  participants  had  difficulty  calling  up  the  ITI-ALC  tech  data.  Some  subsections  were  hundreds  of 
pages  of  PDF  files,  which  took  an  inordinate  time  to  load.] 
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Concept  02  (n=8) 


1  2 

Yes  No 


3.  An  ITI-ALC  system  would  make  estimates  of  task  time  to  complete 


Concept  Q3 

(n=9) 

Q  - 
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u  n 
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3 

4 

5 

More  accurate 

Less  accurate 

4.  If  an  ITI-ALC  system  were  available,  would  you  draw  with  it? 
Mechanics 

Would  use  the  picture  capability  more  than  the  drawing  capability 
Engineers 

As  a  supervisor  maybe 
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5.  Would  an  ITI-ALC  system  be  easy  for  a  novice  maintenance  technician  to  use? 
Inspectors 

As  easy  as  it  is  for  the  inspector 
With  practice 
Mechanics 

Had  never  seen  before  yesterday 
If  familiar  with  computers 


6.  Performing  tasks  outside  your  specialty  with  an  ITI-ALC  system  would  be 
Inspectors 

I-button  makes  it  impossible 
Mechanics 
If  dual  skill 
Don  ’t  know 


and  no  computer  experience 


Concept  Q5  (n=8) 


Yes  No  Maybe 
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Concept  Q6  (n=9) 


Possible  Impossible 


7.  If  you  had  an  ITI-ALC  system,  would  you  use  it  for  your  primary  job? 
Inspectors 
If  more  training 
Mechanics 

Engineering  assistance ,  part  numbers 
For  writing  202s 

If  I  needed  to  pull  up  information  to  do  job 


Concept  Q7  (n=11) 


Yes  No  Depends 


8.  If  an  ITI-ALC  system  were  available  to  you,  you  would  use  it 
Mechanics 

IfTOs  were  changes.  Definitely  for  202s 
202s  and  part  numbers 

Currently  we  use  blueprints,  parts  kits,  and  have  all  paperwork  needed 
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Concept  Q8  (n=11) 


Never  Frequently 


9.  Are  there  maintenance  areas  where  an  ITI-ALC  system  would  be  useful?  If  yes,  which  ones? 
Inspectors 

Other  weapon  systems. 

All  depot  maintenance  on  aircraft,  parts  being  ordered,  E&I,  information  to  technicians,  etc. 

Cockpit.  Tail  to  horizontal  stabilizer 

Mechanics 

202s,  TOs,  Ordering  parts 

Areas  where  a  repair  in  the  TO  can’t  be  found 

All  depot  level  maintenance  tasks,  because  of  being  able  to  access  TOs.  Have  to  access  TOs  first. 
Part  numbers,  repairs,  general  information — TOs 
AC  maintenance  mechanics  could  use  in  place  of  written  TO. 

Engineers 

All 

Documentation,  tracking  and  trending  of  assistance  requests 


Concept  Q9  (n=10) 


.I tiiis 

mm 

Slit 

flfl 

1  2 


Yes  No 


10.  Are  there  maintenance  areas  where  an  ITI-ALC  system  would  NOT  be  useful?  If  yes,  which  ones? 
Inspectors 
Fuel  tank 
Mechanics 
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Tight  areas 

In  place  of  large  blueprints  such  as 


ITI-ALC’s  Perceived  Effect  on  Job  Performance 

1 .  Obtaining  information  using  the  ITI-ALC  system  as  compared  to  current  TOs  was 
Inspectors 
Familiar 
Mechanics 

Easier  depends  on  how  easy  computer  is  to  obtain 


Job  Performance  Qla 
(n=9) 


Job  Performance  Qlb 
(n— 9) 


Faster  Slower  Same 


Easier  Harder  Same 


2.  Obtaining  forms  using  the  ITI-ALC  system  as  compared  to  the  current  process  for  obtaining  forms 
was 

Mechanics 

Prefilled  information  is  helpful 


Job  Performance  Q2a 
(n=9) 
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3.  Ability  to  access  other  types  of  information  (e.g.,  parts  information)  using  the  ITI-ALC  system  as 
compared  to  the  current  method  was 
Inspectors 

Didn  H  access  TOs  or  parts 


Job  Performance  Q3a 
(n=9) 
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Faster  Slower  Same 


Job  Performance  Q3b 
(n=9) 

9 
8 
7 
6 
5 
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3 
2 
1 
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Easier  Harder  Same 


— ■ — i  ■  i 

1  2  3 


4.  Compared  to  the  current  process,  ordering  parts  with  ITI-ALC  would  be/was 
Inspectors 
Hard  to  say 
Familiar 


Faster  Slower  About  same 


Easier  Harder  Same 


5.  Time  required  to  enter  information  into  the  ITI-ALC  system  as  opposed  to  the  current  method  was 
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Mechanics 
Extra  information 


Job  Performance  Q5 
(n=11) 


Longer  Shorter  Same 


6.  To  accomplish  most  tasks,  the  digital  camera  provided 
Mechanics 

Jumps  a  lot  better  than  last  time 

Getting  image  where  you  wanted  it.  Keeping  at  same  distance  while  focusing 

Will  take  picture  of  where  it  is.  Use  a  magnifying  glass  normally.  Zoom  in  may  not  give  all  the 

detail  you  need 

Needs  to  be  autofocus 

Engineers 

Camera  may  not  be  sufficient  for  all  cases.  Pictures  were  out  of focus.  The  pictures  need  to  be 
tied  to  the  aircraft  location. 


Job  Performance  Q6 
(n=8) 


Inadquate  detail  Adequate  detail 
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7.  If  you  had  digital  photos  of  damaged  areas,  the  number  of  trips  to  the  aircraft  to  view  the  damage 
would 
Engineers 

If  the  engineer  could  request  further  clarification  it  would  decrease  the  number  of  trips  even 
further 

[Engineers  wanted  to  be  able  to  request  further  clarification  from  the  mechanic,  allow  the  mechanic  to 
update/modify  the  202A  and  resubmit] 


Job  Performance  Q7 


(n=2) 

n®8tr 

i  .  * '  *  ' s. "  i/ 
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llltt 
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1  "  . . 
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2 

Remain  the  same  Decrease 


8.  Use  of  the  I-button  would  make  the  signature  approval  time 
Inspectors 

Don ’t  know  what  to  say  about  it 
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9.  In  comparison  with  the  current  signature  security  process,  the  I-button  would  make  signature 
approval  security 
Mechanics 

Could  be  lost  like  anything 


Job  Performance  Q9 
(n=9) 


Less  secure  More  secure 


10.  Compared  to  the  current  system,  time  required  to  send  information  using  the  ITI-ALC  system  was 
Mechanics 

Could  be  several  days 


Job  Performance  Q10 
(n=11) 


IS 

iti 
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l-^-l : 

1  2  3 

Longer  Shorter  Same 


1 1 .  While  carrying/wearing  the  system,  it  interfered  with  the  maintenance  task 
Inspectors 
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Very  little 
Mechanics 

Filling  out  202.  Would  take  the  computer  off  for  maintenance 
If  forms,  then  it  did  not  interfere 
I  don  V  think  it  would  affect  it  much 
Depends  on  where  on  A/C 


Job  Performance  Q11 
(n=9) 


Infrequently  Frequently 


12.  As  compared  with  paper,  information  presented  on  the  ITI-ALC  system  was 
Engineers 

Have  to  adjust  to  format 


Job  Performance  Q12 
(n=11) 


1  2  3 


Easier  Harder  Same 
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ITI-ALC  Hardware  and  Software  Use 


1 .  Carrying/wearing  the  ITI-ALC  system  while  performing  your  task  was 
Inspectors 
Not  used  to  it 

Worried  about  banging  it  up.  Would  put  it  down  before  performing  maintenance  inspection 


Usability  Q1  (n=9) 


Convenient  Inconvenient 


2.  Was  the  system  comfortable  to  wear? 
Mechanics 
Awkward 


Usability  Q2  (n=9) 
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Yes  No 
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3.  Did  the  straps  fasten  the  ITI-ALC  system  securely  to  your  body? 
Mechanics 

Had  some  trouble  keeping  it  horizontal  to  the  ground 


4.  While  using  the  computer  to  perform  your  task,  the  use  of  the  pen  was 
Inspectors 
Tapping  was  hard 
Mechanics 
Angle 

Getting  hourglass  to  come  up 

Tapping  was  problematic 

Tapping  and  touching  caused  problems 

[With  few  exceptions,  participants  could  not  tap  the  screen  and  have  the  system  consistently  respond.] 
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5.  Was  it  obvious  when  the  computer  was  busy  working? 


Usability  Q5  (n=11) 


Yes  No 


6.  Information  presented  on  the  ITI-ALC  system  was 
Mechanics 
Better  than  before 
Engineers 

Some  text  was  too  small;  however,  if  it  were  on  a  1 7”  monitor,  it  might  be  OK. 


Usability  Q6  (n=11) 


Easy  to  read  Difficult  to  read 


7.  Information  you  entered  into  the  ITI-ALC  system  was 
Engineers 
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Some  text  was  too  small;  however,  if  it  were  on  a  1 7”  monitor,  it  might  be  OK. 
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Easy  to  read  Difficult  to  read 

8.  Information  presented  on  the  ITI-ALC  system  was 
Mechanics 

TOs  only  (inadequate) 


Usability  Q8  (n=11) 


1  2  3  4  5 

Adequate  Inadequate 


9.  Entering  text  using  the  on-screen  keyboard  was 
Inspectors 

Pretty  easy,  but  need  to  get  used  to  handwriting 
Mechanics 

Better  than  March  demonstration.  Prefer  to  type  information  in  with  on-screen  keyboard 


Ilk 


Not  too  familiar  with  it 


Usability  Q9  (n=9) 


Easy  Difficult 


10.  The  meaning  of  the  “stamp”  icon  was 

If  it  was  not  obvious,  what  would  you  suggest? 


Usability  Q10  (n=3) 


Obvious  Not  Obvious 


1 1 .  The  meaning  of  the  “envelope”  icon  was 
Inspectors 
Once  I  learn  it 

If  it  was  not  obvious,  what  would  you  suggest? 
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Usability  Q11  (n=3) 

3 

2 

1 

0 

Obvious  Not  Obvious 


12.  The  navigation  bar  on  the  right  side  of  the  computer  was 


Usability  Q1 2  (n=9) 


1  2  3  4  5 

Easy  to  understand  Difficult 


13.  Entering  sketches  on  the  ITI-ALC  system  was 
Mechanics 
Pen 
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Usability  Q1 3  (n=8) 


Easy  Difficult 


14.  Using  the  camera  to  photograph  the  damaged  area  was 
Mechanics 
Hard  to  keep  still 
Needs  to  be  autofocus 

Real  sensitive  to  height  and  focus.  Focus  was  slow. 


Usability  Q14  (n=6) 


Easy  Difficult 


15.  Reviewing  photos  you  had  taken  was 
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Usability  Q1 5  (n=6) 


Did  you  notice  when  photos  were  available  for  you  to  review 


Saving  photos  on  the  computer  was 


Usability  Q17  (n=6) 


Easy  Difficult 


1 8.  Viewing  photos  was 


Usability  Q1 8  (n=8) 


Easy  Difficult 


19.  Recording  voice  notes  was 
Mechanics 

Get  somewhere  quiet.  Background  noise  was  a  problem.  A  lot  of  static,  especially  with  mule 
running 
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Usability  Q21  (n=6) 


Transmitting  information  for  other  people  to  read  or  approve  was 
Mechanics 
Easy  to  send 


Usability  Q23  (n=11) 


The  ITI-ALC  system  was 


Usability  Q25  (n=11) 


Not  Useful  Very  Useful 


26.  What  did  you  like  most  about  the  ITI-ALC  system 
Inspectors 

1-button 

Cuts  out  paperwork,  saves  time,  more  secure,  All  information  needed  at  hand 
It’s  not  heavy  to  hold  while  doing  the  job — older  mechanics  need  more  time  working  on  this 
system 

Mechanics 

Fairly  simple  to  operate 
It  makes  202 ’s  faster 
Faster  and  easier  than  paperwork 
Ease  of  filling  out  the  202s 

Was  having  hands  on  information  at  my  finger  tips,  without  having  to  leave  the  job. 

Not  having  to  search  through  TO  to  find  information 
Engineers 

Photo/voice  information 

Complete  and  consolidated  history  of  request 


27.  What  did  you  like  least  about  the  ITI-ALC  system 
Inspectors 
Pen 

It’s  not  in  use  yet!! 

The  glare  on  the  face  made  it  hard  to  see 
Mechanics 

Using  the  pen  for  tapping 
Taking  pictures 
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28. 


The  way  you  access  different  areas  of  the  airplane  for  TO  reference 

Looking  up  information  from  the  TOs.  Too  slow  and  too  hard  to  go  from  page  to  page 

When  you  pulled  up  a  view  of  an  area  it  didn  7  have  the  TO  figure  and  index. 

Not  being  familiar  with  its  use 
Engineers 
Finding  TO’s 
Nothing  stands  out 


Did  you  use  an  ITI-ALC  system  in  a  previous  study? 


Usability  Q28a  (n=10) 
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Remember 

If  yes,  which  one  (screen  presentations  primarily)  did  you  prefer? 


284 


Usability  Q28b  (n=8) 


This  system  Previous  system 
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APPENDIX  F 

SCREEN  WALKTHROUGHS 
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Appendix  F:  Screen  Walkthroughs 


ITI-ALC  Screen  Walkthrough  -  Inspectors 

Login 

User’s  manual  is  needed  to  show  how  to  get  to  it. 

[If  the  I-button  was  not  removed  after  accessing  the  application,  if  frequently  came  loose  and 
dropped  on  the  floor] 


Please  Insert  Your  Electronic  Stamp  Into  The  Receptor. 


287 


Select  Aircraft 

Pick  area  first.  Cards  in  numerical  sequence 

Need  to  assure  that  planes  on  screen  really  represent  what  is  on  the  floor  especially  for  inspector. 
Needs  to  be  kept  up  to  date.  We  can  deep  it  up  to  date. 


Dock  3 


Select  Aircraft 


Dock  2 


J* 

^44  8400056  8600073 

08/18/98 


7900073 

07/20/98 

7900077 : 
07/24/98 

8000056 

08/06/98 


8100051  &■ 
08/11/98 

8000051 : 
07/30/98 


7900074 

07/21/98 

8000053 

08/03/98 

8000057 

08/07/98 


-8000059  >4% 

08/10/98 


8000050 
07/29/98 


7900075 

07/22/98 

8000054 

08/04/98 

8200054 

08/16/98 

8200053 

08/13/98 

8000058 

08/09/98 

7900079 

07/28/98 


7900076 

07/23/98 

8000055 

08/05/98 

8300056 

08/17/98 

8200052 

08/12/98 

8000052 

07/31/98 

7900078 

07/27/98 


Dock  4 


Dock  1 
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173  Task  List 

Pick  area  first.  Cards  in  numerical  sequence.  Everything  OK. 


User:  DAN  El  -  '  Tail  Num:  7900073 

IIP 

r-Hanflor 

tie  trv/  ■ 

Tap  an  area  number  or  scroll  through  this  list  to  view  work  cards. 

Tap  the  circle  icon  to  the  right  of  each  task  to  select  it. 

To  view  the  list  of  all  checked  task,  tap  the  signoff  icon. 

— 

tip; 

T173M«: 

m 

SlgnOff  : 

L 

AFMC  173  Task  List . 

^vHistoryt  ; 

01  -  Forward  Fuselage,  Radome  and  Speed  Brake 

12621 

Visually  inspect  the  left  side  load  scissors  for  cracks,  wear  and  corrosion.  IAW 1 F- 
1 5A/C/E-3-4. 

i|D«foctoSi 

-  ' 

12622 

Visually  inspect  the  right  side  load  scissors  for  cracks,  wear  and  corrosion.  IAW  1 F- 
1 5A/C/E-3-4. 

LJ 

S' 

; 

12643 

Visually  inspect  radar  antenna  for  hydraulic  leaks,  cracks,  corrosion  and  any  other 
defects.  Defects  will  be  annotated  on  AFLC  Form  1 73. 

# 

fflpj 

12657 

Inspect  radome  hinge  and  radome  hinge  back  up  angle  for  warping,  cracks, 
corrosion  and  missing  fasteners  IAW  1 F-1 5A/E-6WC-6  and  1 F-1 5A/C/E-3-1 .  (1 200 
hour  inspection). 

O 

ipt 

12660 

Visually  inspect  upper  and  lower  fuselage  splice  area.  F.S.  41 5,  for  cracks  and  loose 

o 

iPIll 

or  damaged  fasteners.  Check  skin  for  dents,  cracks  and  buckles.  (1 200  hour 
inspection). 

||§ip 

12666 

Visually  inspect  the  left  and  right  diffuser  ramp  fixed  hinge  supports,  using  a  mirror 
and  flashlight,  for  cracks.  Check  the  servocylinder  for  leakage,  corrosion  and  cracks. 
(1200  hour  inspection). 

m 

1  r‘"  ■ 

12684 

Visually  inspect  aileron  rudder  interconnect  (ARI)  for  leakage.  Inspect  support 
bracket  for  cracks  and  scurity.  Inspect  electrical  connectors  for  chaffing  and 

§U 

it 

life 

8| 

L  •  *  j'Ai : '  .2  vfcV  .  r 
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173  Sign-Off  List 

Sign  off  icon  is  like  signing  off  that  procedure.  History  tell  what  has  been  done 
[Inspectors  exhibited  confusion  between  signing  off  using  the  E&I  icon  in  the  top  left  and  the 
‘SignOff’  icon  on  the  right  side  menu  bar.  Typically,  they  picked  the  “SignOff’  icon  instead  of  the  E&I 
icon.] 


Elil 


[Hangar' 

m 

pH 

K 


Tap  an  area  number  or  scroll  through  this  list  to  review  the  checked  tasks . 

To  add  or  remove  a  task  from  this  list  return  to  the  1 73  list. 

To  complete  the  Sign  Off  for  this  list  of  tasks,  plug  in  your  electronic  stamp  and  tap  the  stamp  icon 
above. 


01  -  Forward  Fuselage.  Radome  and  Speed  Brake 


12231 


12305 


1 2622 


[Visually  inspect  the  left  and  right  bypass  air  doors  for  obstsructions,  distortion  and  cracks 
(Door  33  L/R).  (1 200  hour  inspections.) 


Visually  inspect  A/C  prior  to  disassembly  for  general  condition  to  determine  obvious 
discrepancies,  deterioration  (structure  paint,  flight  controls,  landing  gear,  etc)  wear,  tear  and 
cleanliness.  - 


Visually  inspect  the  right  side  load  scissors  for  cracks,  wear  and  corrosion.  IAW 1 F- 
15A/C/E-3-4.-  / 


12643  (Visually  inspect  radar  antenna  for  hydraulic  leaks,  cracks,  corrosion  and  any  other  defects. 
Defects  will  be  annotated  on  AFLC  Form  1 73. 


173  History  List 

Need  to  list  work  category 


User:  DANE! 

01 


’  TailN  urn  :7900G73 

: .y’ '  £v£sy;  .  !:■  ?££ l  A\  f.;v 


T ap  an  area  number  or  scroll  through  this  list  to  review  all  Signed  Off  tasks  for  this  aircraft. 
You  may  tap  an  icon  to  the  right  to  go  to  another  screen  or  exit. 


AFMC  1 73  History  List 

01  -  Forward  Fuselage.  Radome  and  Speed  Brake 

12232 

Visually  inspect  the  left  and  right  bypass  air  louvers  and 
screens  for  obstructions,  cracks  and  FOD.  (1200  hour 
inspection.) 

E&l  Mechanic  II 

06/30/98 

20:13:26 

06  -  Right  Wing 

12207 

Visually  inspect  the  right  wing-to-fuselage  pins  and  bolts  and 
the  fuselage  attach  lugs  for  cracks.  This  satisfies  IAT 1 6. 
Complete  afto  form  3  and  submit  to  LFEFS.  IAW 1 F-1 5A-36. 

E&l  Mechanic  II 

06/30/98 

20:13:26 

12618 

Visually  inspectthe  right  wing  to  fuselage  attach  pins  at  F.S. 
509.5  thru  626.9,  P/N  68A1 1 21 77  and  68A1 1 21 78.  for 
corrosion,  wear,  galling,  fretting  or  any  other  unsatisfactory 
condition.  IAW  1  F-1 5A-36. 

E&l  Mechanic  II 

06/30/98 

20:13:26 

1 2620 

Visually  inspect  the  right  wing  to  fuselage  attach  bushings  at 
F.S.  509.5  thru  626.9.  P/N  68A1 12177  and  68A1 1 21 78.  for 
icracks.  wear,  galling,  fretting  or  any  other  unsatisfactory 
condition.  IAW  1  F-1 5A-36. 

E&l  Mechanic  II 

!  .  '• 

06/30/98 

20:13:26 

Standard  Defect  List 

Looks  good.  Some  need  to  be  added  to  master. 

Not  used  to  using  areas  so  didn ’t  pick  by  area  icon  here.  Record  number  for  deficiencies  column 
header,  not  obvious. 


Tap  an  area  number  or  scroll  through  this  list  to  view  the  defects. 
To  Add  or  Change  a  defect  to  submit  tap  the  related  copy  icon. 


01  -  Forward  fuselage.  Radome  and  Speed  Brake 


FWD  END  OF  HINGE  CRACKED  ON  AIRCRAFT  ABOVE  DOOR  3L 


14  MILD  CORROSION  ON  DOOR  OPEN  LIMIT  SWITCH.  F.S.  460 


15  MILD  CORROSION  ON  DOOR  CLOSED  LIMIT  SWITCH.  F.S.  475 


16  MILD  CORROSION  ON  DOOR  OPEN  AND  CLOSE  LI  M IT  SWITCH  ES  IN  AIR 
REFUELING  COMPARTMENT 


2  FWD  EN  D  OF  HI  NG  E  CRACKED  ON  AI  RCRAFT  ABOVE  DOOR  3R 
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Submit  Defects  List 
Columns  need  labels 

[Participants  exhibited  some  difficulty  remembering  how  to  send  the  defect  list  (the  envelope  icon  in 
the  top  left).  In  several  instances  they  forgot  to  send  it.] 

[Navigation  among  several  screens  was  problematic.  Once  this  screen  was  completed,  the  system 
navigated  back  to  the  1 73  Task  List.  This  automatic  navigation  was  “too  far  back”  and  confusing  to  the 
user.] 


User:  DAN  El 

imBSI 


Tajl  ;Num:  7900073 


lifengarg 


Tap  an  area  number  or  scroll  through  this  list  to  review  recorded  defects. 
Tap  Submit.  Hold,  or  Remove  icon  for  each  defect. 

When  all  defects  have  been  reviewed,  tap  the  send  icon  above. 


.  -  ...  _  ...  Submit  Defects  List  •  _ 

7-  ?,W 

[01  -  Forward  Fuselage.  Radome  and  Speed  Brake 

195157 

| 

FWD  END  OF  HINGE  CRACKED  ON  AIRCRAFT  ABOVE  DOOR  3L . 

IAW  1 F-1 5A-3-2 

$  Submit 

#  Hold 

i£  Remove 

95160 

FWD  END  OF  HINGE  CRACKED  ON  AIRCRAFT  ABOVE  DOOR3L. 

IAW  1  F-1 5A-3-2 

Q>  Submit 

#  Hold 

4  Remove 

9516? 

MILD  CORROSION  ON  DOOR  OPEN  AND  CLOSE  LIMIT  SWITCHES  IN 
AIR  REFUELING  COMPARTMENT,  IAW  1  F-15A-32 

^Submit 

<£  Hold 

$  Remove 

fi 

ispl 
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Defect  History  List 

OK 


Tap  an  area  number  or  scroll  through  this  list  to  review  all  submitted  defects  for  this  aircraft. 
You  may  tap  an  icon  to  the  right  to  go  to  another  screen  or  exit. 


02-  Forward  Cockpit 


95146 

MILD  CORROSION  IN  COCKPIT  ON  FLOOR  UNDER 

PI  LOT'S  CONSOLE  AND  AROUN  D  FASTENERS .  IAW  1  F- 

E&J  Mechanic 
II 

06730798 

18:40:13 

1 5A-23  •' v;:.. 

95147 

MILD  CORROSION  IN  COCKPIT  ON  FLOOR  UNDER 

PILOT'S  CONSOLE  AND  AROUND  FASTENERS .  IAW  1 F- 

E&l  Mechanic 
II 

06/30/98 

18:43:44 

15A-23  -  • 

2A-  Rear  Cockpit  or  Fr  15  Equipment  Computer 

95148 

ADFG  KKKHTR.  IAW  1  FI  5A/C/E-2-28GS-00-1 

E&l  Mechanic 
II 

06/30/98 

18:47:55 

/A 


ACCEPTANCE  AND  FUNCTIONAL  CHECK  FLIGHT  PROCEDURES  f1F-15E-6CF-11 
ACCEPTANCE  AND  FUNCTIONAL  CHECK  FLIGHT  fl  F-1 5E-6CL-11 

ACCESSORY  GEARBOX  SYSTEM  -AIRFRAME  MOUNTED  ACCESSORY  DRIVE  S/S/SN  83-20-01  THRU  83-20- 
30  -  JOB  GUIDE  H  F-1 5E-2-83  JG-20-11 

ACCESSORY  GEARBOX  SYSTEM  -AIRFRAME  MOUNTED  ACCESSORY  DRIVE  S/S/SN  3-20-31  THRU  END  - 
JOB  GUIDE  H  F-1 5E-2-83  JG-20-21 

ACCESSORY  GEARBOX  SYSTEM  -DRIVESHAFT-  JOB  GUIDE  n  F-1 5E-2-83JG-1 0-11 

ACCESSORY  GEARBOX  SYSTEM  -GENERAL- JOB  GUIDE  fl F-1 5E-2-83 JG-00-1 1 

AEROSPACE  GROUND  EQUIPMENT  -  IPB  fl  F-1 5E-4-S1 

AIR  CONDITIONING  SYSTEM  -DISTRIBUTION  -  JOB  GUIDE  flF-1 5E-2-21 JG-21-11 

AIR  CONDITIONING  SYSTEM  -  FAULT  ISOLATION  (1  F-1 5E-2-21  FI-00-11 

AIR  CONDITIONING  SYSTEM  -  GENERAL  SYSTEM  nF-15E-2-21GS-00-11 

AIR  CONDITIONING  SYSTEM  -GENERAL- JOB  GUIDE  0  F-1 5E-2-21 JCF00-11 

AIR  CONDITIONING  SYSTEM  -LIQUID  COOLANT  -  JOB  GUIDE  HF-15E-2-21 JG-80-11 

AIR  CONDITIONING  SYSTEM  -PRESSURIZATION  CONTROL  -ANTI-G  -  JOB  GUIDE  flF-1 5E-2-21 JG-33-11 

AIR  CONDITIONING  SYSTEM  -PRESSURIZATION  CONTROL  -  CABIN  -  JOB  GUIDE  MF-1 5E-2-21 JG-30-11 

AIR  CONDITIONING  SYSTEM  -PRESSURIZATION  CONTROL -CANOPY  SEAL  -  JOB  GUIDE  HF-15E-2-21 JG- 

31-11 

AIR  CONDITIONING  SYSTEM  -PRESSURIZATION  CONTROL -WAVEGUIDES  -  JOB  GUIDE  fl  F-1 5E-2-21 JG-32- 

1) 

AIR  CONDITIONING  SYSTEM  -  SCHEMATIC  DIAGRAM  HF-15E-2-21 SD-00-11 

AIR  CONDITIONING  SYSTEM  -TEMPERATURE  CONTROL- AIR  CYCLE  AIR  CONDITIONING  SYSTEM  S/S/SN 
21-64-01  THRU  21-64-21  -  JOB  GUIDE  (1F-15E-2-21 JG-E4-11 

AIR  CONDITIONING  SYSTEM  -TEMPERATURE  CONTROL  -AIR  CYCLE  AIR  CONDITIONING  SYSTEM  S/S/SN 
21-64-22  THRU  END  -  JOB  GUIDE  fl  F-1 5E-2-21 JG-6< 4-21 

AIR  CONDITIONING  SYSTEM  -TEMPERATURE  CONTROL -AVIONICS  -  JOB  GUIDE  f1F-15E-2-21  JG-66-11 
AIR  CONDITIONING  SYSTEM  -TEMPERATURE  CONTROL -BLEED  AIR-  JOB  GUIDE  f1F-15E-2-21  JG-6D-11 
AIR  CONDITIONING  SYSTEM  TEMPERATURE  CONTROL  -  CABIN  -  JOB  GUIDE  HF-15E-2-21 JG-65-11 
AIRCRAFT  AND  EQUIPMENT  -  LIST  OF  APPLICABLE  PUBLICATIONS  i1F-15A-011 
AIRCRAFT  BATTLE  DAMAGE  REPAIR  fl  F-1 5E-391 

AIRCRAFT  DESCRIPTION  AND  MAINTENANCE  ORIENTATION  -  GENERAL  VEHICLE  MF-15E-2-00GV-00-11 
AIRCRAFT  -ENGINE  AND  JFS  OPERATION  S/S/SN  05-20-01  THRU  05-20-07  -  JOB  GUIDE  fl  F-15E-2-05JG-20-11 
AIRCRAFT -ENGINE  AND  JFS  OPERATION  S/S/SN  05-20-08  THRU  END  -  JOB  GUIDE  fl  F-1 5E-2-05  JG-2Q-21 
AIRCRAFT -GENERAL  MAINTENANCE  S/S/SN  05-00-01  THRU  05-00-11  -  JOB  GUIDE  fl  F-1 5E-2-05JG-00-11 
AIRCRAFT -GENERAL  MAINTENANCE  S/S/SN  05-00-1 2  THRU  05-00-23-  JOB  GUIDE  f1F-15E-2-05JG-0Q-21 
AIRCRAFT  -GENERAL  MAINTENANCE  S/S/SN  05-00-24  THRU  END  -  JOB  GUIDE  nF-15E-2-05JG-00-31 
AIRCRAFT -EMERGENCY  PROCEDURES  -  JOB  GUIDE  fl  F-1 5E-2-05  JG-30-11 
AIRCRAFT  LAUNCH  INSPECTION  PROCEDURES  -  WORKCARDS  fl  F-1 5E-6WC-2-21 


Defect  Fill-In 

Handwriting  is  alright.  Needs  to  pick  up  writing  a  little  better. 

[Recording  the  defect  by  pressing  the  +  icon  was  problematic.  If  the  participant  pressed  “Submit 
Defects”  icon  from  the  right  menu  (a  common  mistake),  a  dialog  box  was  presented.  This  dialog  box 
was  designed  so  that  if  it  was  accepted,  the  data  entered  on  this  form  was  lost,  if  the  dialog  was  rejected 
the  participant  had  a  second  chance  to  press  the  +  icon.  This  screen  interface  (e.g.,  the  +  icon)  was  quite 
confusing  and  caused  numerous  errors  along  with  unnecessary  re-entry  tasks  due  to  the  dialog] 


Add  or  Change  all  information  needed  to  complete  this  defect. 

When  the  defect  is  complete,  tap  the "+"  icon  to  record. 

Area 

1 01  -  Forward  Fuselage.  Radome  and  Speed  Brake 

T.O.  Number 

1 F-1 5A-3-2 

d 

Skill  Code 

AS  -  Aircraft  Sheetmetal 

. . . . . d 

Work  Unit  Code 

1 1 A000  -  Forward  Section 

-  V 

HOWMAL  Code 

190 -Cracked. 

.  . . -1 

Description 

FUD  END  OF  HINGE  CRACKED  ON  AIRCRAFT  ABOVE 

DOOR  3L  d 

_ _ I: 

£ 


tm 


Wmi 
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-Ifljxl 
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ITI-ALC  Screen  Walkthrough  -  Mechanics 

Select  Aircraft 

Tail  number  is  good.  Output  date  is  helpful. 

Easy  to  understand 
AFMC  Form  202  Part  A 

Easier  than  our  202s.  Indicators  for  picture  and  voice  were  small,  but  adequate 


IV 

Select  Aircraft 

Dock  3 

Dock  2 

^SM^chl' 

'  'fC*  5 

7900073 

07/20/98 

■  7900074 

07/21/98 

7900075 

07/22/98 

7900076 

07/23/93 

B 

L  8 

7900077 

07/24/98 

8000053 

08/03/98  X''"v' 

8000054 

03/04/98  ^ 

8000055 

08/05/98 

US-’. 

, 

8000056 

*Jfar-  8000057  'tyfisi.. 

8200054 

8300055 

IVIV 

08/06/98 

08/07/98  4^“ 

08/16/98 

08/17/98 

'  ;  \  ,s  V  V  < 

8400056 

08/18/98 

8500073 

08/19/98  .-4*"'-'* 

8200053 

08/13/98 

.  8200052 
08/12/98 

511®  IS 

8100051 
08/1 1/98 

1.8000059 

4*^'  08/10/98  " 

8000058 

08/09/98 

8000052 

07/31/98 

’iffM.',- 

8000051 

,,4.,.  8000050 

7900079  &i4'. 

7900078 

07/30/98 

07/29/98 

07/28/98  / 

07/27/93 

Dock  4  Dock  1 
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Form  202  Part  A 
Pretty  good 

[Several  participants  tried  to  use  the  “T.O.”  icon  on  the  right  menu  bar  to  fill-in  the  drop-down  entitled 
“Tech  Order.”] 


AIMC  Form  202  Part  A 

Tap  the  parts  icon  to  choose  a  part  number  from  the  aircraft  drawing.  Tap 
the  sketch  or  sound  buttons  to  add  these  features  to  your  request.  When 
Form  202  is  complete,  tap  the  send  button  to  submit  it  to  an  engineer. 


Part  Number 


Tech  Order 


"3 


i  *******■***••*•*******■*■*•*•***** 


Select  One 


Work  Stoppage 


Yes^  Organically  Caused  No 


Deficiency  &  Recommendations 


TroyAS  ' 
7900073 
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Thank  You.  YourAFMC... 
Satisfied  with  that 


IVrhankyou.  Your  AFMC  Form  202  A  has  been  sent  with  the  following 
Control  Number:  7900073-0087 


To:  Engineering 

Initiator:  Troy  Gould 
Office  Symbol:  LFEFS 
P  h  o  ne:  9 1 2-926-5407 

Location:  Bldg.  300 

Air  Logistic  Specialist:  Jan  Felcan 
Office  Symbol:  LFPB 
Phone:912-926-6030 

Deficiency  &  Recommendations 

MULTIPLE  RIVETS  MISSING  ON  LEFT 
BOTTOM  SIDE  OF  NOSE.  REQUEST 
ENGINEERING  ASSISTANCE 


Date:  8/13/1 998 

Control  Number:  7900073-0087 
Tail  Number:  7900073 

Part  Number:  68A230 173-2037 
Part  Description:  POD  ASSY,  Vertical 
stabilizer  2.00  diameter  tip 

NSN:  3333-000003 
TO/Dwg  Number  1 F-15E-3-3 

Work  Stoppage:  Yes 
Organically  Caused:  No 


Tap  the  Start  button  to  fill  out  a  new  AFMC  Form  202  Part  A 
for  Tail  Number:  7900073 


nn 
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Below  is  a  list  of  all  form  202’s  you  have  submitted 

Fuselage  station  could  be  used  to  help  identify/differentiate  between  202s.  Noun  would  be  good 
column  to  help  differentiate  among  202s 
Fine 


Bjlow  is  a  list  of  all  the  Form  202's  you  have  submitted  and  their  current  — 

TroyAS 

status. 

iioisKi 

Tap  the  view  button  to  see  the  details  of  a  Form  202 

mm 

Control 

Number 

Part  Number 

Date 

Status 

ViewAEMC 
Form  202 

7900073-0048 

68A230 174-2005  08/05/98 

Complete 

View 

*^202^ 

7900073-0049 

HT4057-3-6A 

08/05/98 

Complete 

View 

7900073-0050 

HT4057-3-6A 

08/05/98 

Complete 

View 

mmm® 

7900073-0052 

SDF  SD 

08/05/98 

Complete 

View 

7900073-0053 

SD  SD  SD 

08/05/98 

Assigned  to  Gaylord 
Oyler 

Work  in  Progress 
(Gaylord  Oyler) 

View  | 

7  HlsUwyf 

SB 

7900073-0054 

DSF  FDHGH 

08/05/98 

View  | 

7900073-0055 

68A230 173-2037  08/06/98 

Complete 

View  j 

m&z. 

7900073-0058 

68A230191-2005  08/06/98 

Complete 

View 

7900073-0060 

SDF  SD 

08/11/98 

In  Scheduling 

View 

7900073-0077 

HT4057-3-6A 

08/11/98 

In  Scheduling 

View 

7900073-0087 

68A230 173-2037  08/13/98 

In  Scheduling 

View 

:  V 

7onnn7^-nins> 

6RA0?niQi-?nn*; 

In  !^rh  prhtlin  or 

Vifiw  1 

i  vi 
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Part  number  selection 

Good  pictures.  Large.  Like  index  picking 

May  need  to  point  to  an  area,  rather  than  the  reference  designator.  In  case  ref  des  does  not  refer 
to  area  damaged. 

Pretty  easy  to  do.  Point  to  part  and  number  comes  up.  Didn’t  show  all  part  #’s.  Need  to  show  all. 
TO  and  Figure  with  this  picture.  This  is  a  lot  of  help  right  here.  TO  figure  and  index  is  needed 
for  ordering  part 
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Below  is  a  list  of  all  the  Form  202's  that  have  been  completed  for  this 

aircraft. 

Tap  the  view. button  to  see  the  Details  of  a  Form  202 


Control  Number 

Initiator 

Part  Number 

Date 

ViewAFMC 
Form  202 

7900073-0007 

AS  Mechanic  I 

68A230 174-2005 

08/04/98 

View  j 

7900073-0048 

Troy  Gould 

68A230174-2005 

08/05/98 

View 

7900073-0049 

Troy  Gould 

HT4057-3-6A 

08/05/98 

View 

7900073-0050 

Troy  Gould 

HT4057-3-6A 

08/05/98 

View 

7900073-0052 

Troy  Gould 

SDFSD 

08/05/98 

View 

7900073-0055 

Troy  Gould 

68A230 173-2037 

08/06/98 

View 

7900073-0058 

Troy  Gould 

68A230191-2005 

08/06/98 

View 

Record  Sound  Note 
Send/Close 
Easiest  part 
Easy  to  use 

When  record  not  a  good  idea  to  do  it  out  there  (on  the  floor).  More  static  than  anything  else. 


AFMC  Form  202  Part  A 

Tap  the  parts  icon  to  choose  a  part  number  from  the  aircraft  drawing.  Tap 
the  sketch  or  sound  buttons  to  add  these  features  to  your  request.  When 
Form  202  is  complete,  tap  the  send  button  to  submit  it  to  an  engineer. 


Part  Number 


e 


Tech  Order 

1F-15E-3-3  Door: 


Record  Sound  Note 


Record  Sound  j 


Stop  Recording 


Work  Stoppage 
Deficiency  &  Recc 


Play  Sound 


Send 


TroyAS 
7900073  v 
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AFMC  Form  202  Review 

202  History  helpful  on  what  else  has  been  written  up 


k  AFMC  Form  202 

After  reviewing  the  Form  202  data  below,  you  may  tap  an  icon  to  the  right  to  go  to  another 

screen  or  exit. 


Part  A 

To  Engineer:  Engineering 

Initiator:  AS  Mechanic  I 
Office  Symbol:  LFPB 
Phone:  912-926-6030 
Location:  Bldg.  S3 

Date:  08/12/98 

Control  Number:  7900073-0083 
Tail  Number:  7900073 

Air  Logistic  Specialist:  Jan  Felcan 
Office  Symbol:  LFPB 
Phone:  912-926-6030 


PartB 

To  Initiator:  AS  Mechanic  I 
From  Engineer/ES:  Engineering 
Date: 

Disposition  Instructions: 

Repair  Instructions 

Recissions  Completion  of  S/N: 
Recissions  Date: 

Requires  AFMC  Form  252: 

Requires  AF  Form  2600: 

AFTO  Form  95/DD  Form  1574  Entry 
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TO  Selection 

Would  like  to  pick  TOs  based  on  physical  location  on  aircraft.  Provide  only  information/TOs 
relevant  to  the  specialty  of  the  technician 


Search  TOs  By  Document  Title 

fl  |  B  |  C  1  D  I  E  j  FTe"!  H  |  T> 


•'1 ,,  “ 


PH 

CE 

J _ 


[ryy-v. 


III  Mcraft  StorageManuat  j||  olBrffi;* .*  g. 


ACCEPTANCE  AND  FUNCTIONAL  CHECK  FLIGHT  PROCEDURES  flF-15E-6CF-11 
ACCEPTANCE  AND  FUNCTIONAL  CHECK  FLIGHT  fl  F-1 5 E-6CL-11 

ACCESSORY  GEARBOX  SYSTEM  -AIRFRAME  MOUNTED  ACCESSORY  DRIVE  S/S/SN  83-20-01  THRU  83-20- 
30  -  JOB  GUIDE  fl  F-1 5E-2-83JG-20-1 1 

ACCESSORY  GEARBOX  SYSTEM  -AIRFRAME  MOUNTED  ACCESSORY  DRIVE  S/S/SN  3-20-31  THRU  END  - 
JOB  GUIDE  fl  F-1 5E-2-83  JG-20-21 

ACCESSORY  GEARBOX  SYSTEM  -DRIVESHAFT  -  JOB  GUIDE  (1  F-1 5E-2-83  JG-1 0-1 ) 

ACCESSORY  GEARBOX  SYSTEM  -GENERAL-  JOB  GUIDE  flF-15E-2-83JG-00-tt 
AEROSPACE  GROUND  EQUIPMENT  -  IPB  AF-15E-4-6) 

AIR  CONDITIONING  SYSTEM  -DISTRIBUTION  -  JOB  GUIDE  fl  F-1 5E-2-21 JG-21  -1 1 

AIR  CONDITIONING  SYSTEM  -  FAULT  ISOLATION  fl  F-1 5E-2-21  FI-00-1 1 

AIR  CONDITIONING  SYSTEM  -  GENERAL  SYSTEM  H  F-1 5E-2-21 GS-00-1 1 

AIR  CONDITIONING  SYSTEM  -GENERAL  -  JOB  GUIDE  fl  F-1 5E-2-21 JG-QQ-1 1 

AIR  CONDITIONING  SYSTEM  -LIQUID  COOLANT-  JOB  GUIDE  AF-15E-2-21 JG-80-11 

AIR  CONDITIONING  SYSTEM  -PRESSURIZATION  CONTROL -ANTI-G  -  JOB  GUIDE  AF-15E-2-21 JG-33-11 

AIR  CONDITIONING  SYSTEM  -PRESSURIZATION  CONTROL  -  CABIN  -  JOB  GUIDE  AF-15E-2-21 JG-30-11 

AIR  CONDITIONING  SYSTEM -PRESSURIZATION  CONTROL -CANOPY  SEAL -JOB  GUIDE  fl  F-1 5E-2-21 JG- 

31-11 

AIR  CONDITIONING  SYSTEM  -PRESSURIZATION  CONTROL -WAVEGUIDES  -  JOB  GUIDE  fl  F-1 5E-2-21 JG-32- 

II 

AIR  CONDITIONING  SYSTEM  -  SCHEMATIC  DIAGRAM  fl  F-15E-2-21SD-00-11 

AIR  CONDITIONING  SYSTEM  -TEMPERATURE  CONTROL-  AIR  CYCLE  AIR  CONDITIONING  SYSTEM  S/S/SN 
21-64-01  THRU  21-64-21  -  JOB  GUIDE  fl  F-1 5E-2-21 JG-64-1 1 

AIR  CONDmONING  SYSTEM  -TEMPERATURE  CONTROL-AIR  CYCLE  AIR  CONDITIONING  SYSTEM  S/S/SN 
21-84-22  THRU  END  -  JOB  GUIDE  fl  F-1 6E-2-21 JG-64-21 

AIR  CONDITIONING  SYSTEM -TEMPERATURE  CONTROL-AVIONICS- JOB  GUIDE  fl  F-1  SE-2-21 JG-66-1 1 
AIR  CONDITIONING  SYSTEM  -TEMPERATURE  CONTROL -BLEED  AIR  -  JOB  GUIDE  AF-15E-2-21  JG-S0-11 
AIR  CONDITIONING  SYSTEM  TEMPERATURE  CONTROL  -  CABIN  -  JOB  GUIDE  fl  F-1 5E-2-21 JG-B5-11 
AIRCRAFT  AND  EQUIPMENT- UST  OF  APPUCABLE  PUBUCATIONS  AF-15A-Q11 
AIRCRAFT  BATTLE  DAMAGE  REPAIR  fl  F-1 5E-391 

AIRCRAFT  DESCRIPTION  AND  MAINTENANCE  ORIENTATION  -  GENERAL  VEHICLE  flF-15E-2-OOGV-OQ-11 
AIRCRAFT -ENGINE  AND  JFS  OPERATION  S/S/SN  05-20-01  THRU  05-20-07-  JOB  GUIDE  f1F-15E-2-0SJG-20-tt 
AfRCRAFT -ENGINE  AND  JFS  OPERATION  S/S/SN  05-20-08  THRU  END  -  JOB  GUIDE  flF-15E-2-05JG-20-21 
AIRCRAFT  -GENERAL  MAINTENANCE  S/S/SN  05-00-01  THRU  05-00-1 1  -  JOB  GUIDE  flF-15E-2-05JG-0Ci-11 
AIRCRAFT -GENERAL  MAINTENANCE  S/S/SN  05-00-1 2  THRU  05-00-23  -  JOB  GUIDE  A  F-1 5  £-2-0  5  JG-0  0-2) 
AIRCRAFT  -GENERAL  MAINTENANCE  S/S/SN  05-00-24  THRU  END  -  JOB  GUIDE  AF-1 5E-2-05JG-00-31 
AIRCRAFT -EMERGENCY  PROCEDURES  -  JOB  GUIDE  f1F-1 5E-2-05JG-30-1) 

AIRCRAFT  LAUNCH  INSPECTION  PROCEDURES  -  WORKCARDS  AF-15E-6WC-2-2) 

Ain^nAtrr  i  n/ci  iKir  ihd  r'l  unc  /i  c  i  cc  o  no  tr  nn  in  . 
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Take  Picture 

Good  if  it’s  general,  but  if  magnification  is  required...  If  need  a  scribe  then  not  enough  for  area. 
Liked  being  able  to  send  pictures 


[Warning:  Applet  Window 
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ITI-ALC  Screen  Walkthrough  -  Engineers 


Assign 

Should  be  able  to  open  the  first  one  and  continue  on  through  all  of  the  new  ones  and  assign  them 


without  going  back  to  main  screen. 


SCREEN  NOT  SHOWN 


Form  202B 

Highlight  the  deficiency  and  recommendation.  Visually  enhance  these  because  refer  to  it  most. 
Did  engineer  answer  the  entire  request?  How  was  the  answer? 


AFMC  Form  202  A 


User:; 


Submit 


202  Selection 

Submit  is  an  unusual  word  choice.  “ Open  ”  might  be  better  word  choice. 

\  AFMC  Form  202  Selection  Screen 

To  fill  out  AFMC  Form  202  Part  B  for  a  particular  item,  click  the  Submit  button. 

Control  Number  Username  Date  Part  Number 

7900073-0053  Troy  Gould  08/05/98  SD.SDSD  Submit  | 

7900073-0054  Troy  Gould  08/05/98  DSFFDHGH  Submit 

7900073-0061  AS  Mechanic  I  08/06/98  68A230 191-2005  Submit 

7900073-0064  AS  Mechanic  I  08/06/98  68A230 173-2037  Submit 

7900073-0065  AS  Mechanic  I  08/06/98  68A230191-2005  Submit 


History 

No  problem 


1^  Select  a  Tail  Number  to  View  its  History  of  AFMC  Form  202 


AFCT  SERIAL  NO 

OUTPUT  DATE 

View  AFMC  Form 
202’s 

7900073 

07/20/98 

Goto  History  | 

7900074 

07/21/98 

Goto  History 

7900075 

07/22/98 

Goto  History 

7900076 

07/23/98 

Goto  History 

7900077 

07/24/98 

•  e^Goto  History 

7900078 

.  07/27/98 

.  Histoiy 

7900079 

07/28/98 

GotoHistory 

8000050 

07/29/98 

Goto  History 

8000051 

I  07/30/98 

Goto  History 

8000052 

07/31J98 

:  Goto  Histoiy 

8000053 

08/03/98 

Goto  History 

8000054 

08/04/98 

Goto  History 

8000055 

08/05/98 

’Goto  History 

8000056 

08/06/98 

^.Gpto^History 

8000057 

08/07/98 

;  3:;:GptD  History 

8000058 

08/09/98 

f  ;  Goto  History 

8000059 

08/10/98 

Goto  History 

8100051 

08/11/98 

:  Goto  History  J 
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ACCEPTANCE  AND  FUNCTIONAL  CHECK  FLIGHT  PROCEDURES  f1F-15E-6_CFr11 
ACCEPTANCE  AND  FUNCTIONAL  CHECK  FLIGHT fl  F-1 5E-6CL-11 

ACCESSORY  GEARBOX  SYSTEM  -AIRFRAME  MOUNTED  ACCESSORY  DRIVE  S/S/SN  83-20-01  THRU  83-20- 
30  -  JOB  GUIDE  fl  F-1 5E-2-83JG-2Q-.il 

ACCESSORY  GEARBOX  SYSTEM -AIRFRAME  MOUNTED  ACCESSORY  DRIVE  S/S/SN  3-20-31  THRU  END- 
JOB  GUIDE  HF-1 5E-2-83JG-2Q-21 

ACCESSORY  GEARBOX  SYSTEM  -DRIVESHAFT-  JOB  GUIDE  fl  F-1 5E-2-33JG-1 0-11 

ACCESSORY  GEARBOX  SYSTEM  -GENERAL-  JOB  GUIDE  fl  F-1 5E-2-33JG-00-11 

AEROSPACE  GROUND  EQUIPMENT- IPBflM5B±6j 

AIR  CONDITIONING  SYSTEM  -DISTRIBUTION  -  JOB  GUIDE  HF-15E-2-21 JG-21-11 

AIR  CONDITIONING  SYSTEM  -  FAULT  ISOLATION  fl  F-1 5E-2-21  FI-00-11 

AIR  CONDrTIONING  SYSTEM  -  GENERAL  SYSTEM  MF-15E-2-21GS-00-1) 

AIR  CONDITIONING  SYSTEM -GENERAL -JOB  GUIDE  fl  F-1 5E-2-21 JG-0Q-11 

AIR  CONDITIONING  SYSTEM  -UQUID  COOLANT-  JOB  GUIDE  f1F-15E-2-21  JG-30-11 

AIR  CONDITIONING  SYSTEM  -PRESSURIZATION  CONTROL -ANTFG  -  JOB  GUIDE  flF-ISE-2-21  JG-33-11 

AIR  CONDITIONING  SYSTEM -PRESSURIZATION  CONTROL -CABIN -JOB  GUIDE  fl  F-15E-2-21 JG-30-1) 

AIR  CONDITIONING  SYSTEM  -PRESSURIZATION  CONTROL -CANOPY  SEAL-  JOB  GUIDE  HF-15E-2-21 JG- 
31-11 

AIR  CONDITIONING  SYSTEM  -PRESSURIZATION  CONTROL  -WAVEGUIDES  -  JOB  GUIDE  HF-1 5E-2-21 JG-32- 


•  AIR  CONDITIONING  SYSTEM  -  SCHEMATIC  DIAGRAM  f1F-15E-2-21SD-DD-11 

•  AIR  CONDITIONING  SYSTEM  -TEMPERATURE  CONTROL- AIR  CYCLE  AIR  CONDITIONING  SYSTEM  S/S/SN 
21-64-01  THRU  21-64-21  -  JOB  GUIDE  HF-15E-2-21 JG-64-1) 

•  AIR  CONDITIONING  SYSTEM  -TEMPERATURE  CONTROL -AIR  CYCLE  AIR  CONDITIONING  SYSTEM  S/S/SN 
21-64-22  THRU  END  -  JOB  GUIDE  flF-ISE-2-21  JG-64-21 

•  AIR  CONDITIONING  SYSTEM  -TEMPERATURE  CONTROL -AVIONICS  -  JOB  GUIDE  C1F-15E-2-21 JG-66-11 

•  AIR  CONDITIONING  SYSTEM  -TEMPERATURE  CONTROL  -BLEED  AIR  -  JOB  GUIDE  flF-ISE-2-21  JG-S0-1) 

•  AIR  CONDITIONING  SYSTEM  TEMPERATURE  CONTROL- CABIN -JOB  GUIDE  fl  F-1 5E-2-21 JG-65-11 

•  AIRCRAFT  AND  EQUIPMENT  -  UST  OF  APPUCAB  LE  PUBLICATIONS  (1F-15A-011 

•  AIRCRAFT  BATTLE  DAMAGE  REPAIR  fl  F-1 5E-331 

•  AIRCRAFT  DESCRIPTION  AND  MAINTENANCE  ORIENTATION  -  GENERAL  VEHICLE  fl  F-15E-2-00GV-0n-11 

•  AIRCRAFT  -ENGINE  AND  JFS  OPERATION  S/S/SN  05-20-01  THRU  05-20-07  -  JOB  GUIDE  f1F-15E-2-05JG-20-11 

•  AIRCRAFT -ENGINE  AND  JFS  OPERATION  S/S/SN  05-20-08  THRU  END- JOB  GUIDE  fl  F-1 5E-2-05JG-20-21 

•  AIRCRAFT -GENERAL  MAINTENANCE  S/S/SN  05-00-01  THRU  05-00-1 1  -  JOB  GUIDE  fl  F-1 5E-2-05JG-00-11 

•  AIRCRAFT  -GENERAL  MAINTENANCE  S/S/SN  05-00-1 2  THRU  05-00-23  -  JOB  GUIDE  fl  F-1 5E-2-05JG-0Q-21 

•  AIRCRAFT  -GENERAL  MAINTENANCE  S/S/SN  05-00-24  THRU  END  -  JOB  GUIDE  f1F-15E-2-05JG-00-31 

•  AIRCRAFT  -EMERGENCY  PROCEDURES  -  JOB  GUIDE  fl  F-1 5E-2-Q5  JG-30-11 

•  AIRCRAFT  LAUNCH  INSPECTION  PROCEDURES  -  WO RKCARDS  fl  F-1 5E-6WC-2-21 

_  AinmAC-r  i  c\/ci  iMr>  ihd  h  line  /ic  i  cc  o  no  ir  nn  i\ 


Sketch  tool 

To  be  able  to  pick  from  other  photos.  Attach  from  other  places. 
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